How to use this sceneForm for AugmentedFace?

[zjafarian]

HI
I develop makeup by augmented face using SceneView, but AugmentedFaceNode is not for rendering, can you help me?

[ThomasGorisse]

Did you try the 1.1.5 AugmentedFace sample ?

[zjafarian]

No, I didn't see or find the 1.1.5 AugmentedFace sample, could you send a link? thank you

[ThomasGorisse]

The 1.1.5 sample is here : v1.15.0/sceneform-android-sdk/samples/augmentedfaces/

@grassydragon @Oleur @issacclee @fvito @issacclee Anyone who want to have fun moving the 1.15.0 AugmentedFaceNode and AngmentedFaces sample to master?

It shouldn't be very difficult but I actually don't know why the AugmentedFaceNode class was removed on the 1.16.0

[ThomasGorisse]

It will be great. Thanks a lot.

[fvito]

I think I've figured out why AugmentedFaceNode is not in public sceneform release. I suspect it's because it uses proprietary code or assets (like the face_mesh and face_occluder_mesh) or they were not able to convert it from sfb.
They also seemed to have removed the SkeletonNode which is an important part of the AugmentedFaceNode (in order to translate the mesh being projected on the face).

So far what I've managed to figure out is, augmented faces example is showing 2 parts.
1.) Projecting mesh on top of the face
2.) Projecting face texture onto the face

At first, I tried to port just the face texture projection, but I'm now a bit skeptical that is possible. That example uses sceneform_face_mesh and sceneform_face_mesh_occluder which are not availabile in the public sceneform repo. I've tried to take the canonical_face_mesh.fbx and convert it to glb, but it seems to be broken, as the submesh count is returned as 0 (which then breaks the material loading) - something that has been reported to their repository.

I'm still attempting to maybe atleast port the mesh projection, but its not as easy as I thought 🙂
From what I understand, I need to take the face regions and apply their position to the bones of the projected model.

[ThomasGorisse]

Can you send me the fbx you found?
I'll try to have a look at what is inside.
If you have any .obj, I'm also interested.

For the bone position there is the ‘FilamentInstance.setBonesMatrix()‘ from what I know. You can maybe have a look at this.

[fvito]

At first I tried using mesh from here as that one looks more like a face. but have now found the "meshes" that I was missing in obj format here but when converting them to glb with blender, I still get submesh error. Although I'm not sure if they are correct, since they seem to just be a simple plane. Also, does the material also get converted or do I need to convert it with matc?

Another obstacle I encountered is, getting the actuall bone/joints from the models. The sample fox_face and the guide on AR Core, expects the assets to be a single mesh with 4 joints. In order to position them correctly, we would need to get them from the file. I tried looking at how the glb is read/loaded but was only able to find a buffer. I assume this is done in the gltfio library. Renderable.getRenderableInstance.getFillamentAssets.Entities does contain them, but I was not able find a way to get the actuall entity, EntityManager only supports creating and destroying them. The only references to any skeleton/bones seems to be related to SFB file that was being used before.

[fvito]

I have some good news.
I was able to make the facemesh material/occluder to work. This means we can atleast project different textures into the face. For the other part of the sample (projecting mesh) I'm still looking at how to support bones.
I've found a sorta way to get the bones via transform manager, will play around with that a bit. Worse case, atleast face texture now works

Also, sorry for spamming here

[ThomasGorisse]

Hi @fvito,

My apologies for not responding sooner I was working on the 1.19.1

So, to summarize, if I well understood, what you only missing is a way to retrieve the bones matrix from the glb file (FilamentInstance) in order to map it with the ARCore face matrix?

Could you make the Pull Request in order to help you with that?

Don't worry by the fact that it is still not finalize. The PR will be Squashed and merged at the end.
That means that it will create a branch on which you can commit all and as much as you want on it and at the merge time, every commit on that branch will become a single commit on the master branch.

Thanks

[fvito]

Hi, no worries 🙂

I've just opened the PR with the ported sample. I the end I did not use the bones, as even with just basic translation/rotation it behaved the same as their sample so I'm not sure we need bone translation.
I'm facing another issue though, where the camera stream stopped rendering when using front facing camera. I've written more about it in the PR description. If anyone has any idea what change might have caused it, I would appriciate any sort of help.