It is difficult to drag the real face

[tsuJohc]

Hi, thanks for implementing the excellent work!
I try to drag the real face from internet but it seems that DragGAN is difficult to move the handle points to target. The similar operation is easily achieved in synthetic face.

Here is the real face.

Here is the process of the drag. The handle points only oscillate around the starting position. The algorithm takes 50 steps. More optimization steps do not make any help.
https://github.com/Zeqiang-Lai/DragGAN/assets/129818160/9f566779-a6ea-4528-a8bf-1ae775f0b492

Is there problem in GAN Inversion? Or this is an inherent flaw in the DragGAN algorithm?

[Zeqiang-Lai]

I think the projective image of GAN inversion is pretty good, and the problem you face is the flaw in the DragGAN algorithm.

[tsuJohc]

Thanks for active reply!

Could you give a more intuitive explanation about "the flaw in DragGAN" ? The algorithm works well in the synthetic face from DragGAN but fails in real face after inversion. Is there any difference between latent code (W space) from synthetic face and latent code from real face inversion?

[Zeqiang-Lai]

Well, I don't think this problem is specific for real image. What I mean is that the DragGAN algorithm could fail even if the image is synthetic by GAN without inversion.

Besides, the latent code of synthetic face and projected face should not make too much difference with respect to the DragGAN, but the projective one might be less accurate to generate a sharp image.

Further, the oscillation might be caused by the similarity of handle points and points around the target positions. So the handle points could not precisely reach the target points.

[tsuJohc]

Thanks for the profound insights! I agree that inverted real image is obscure and lose a lot of details.

However, I came across the situation that I drag the eyebrow up where the similarity of source points and the target points are quite large.
I dump the curve of motion loss (i.e., FF.l1_loss(f2,f1) ) for 3 cases:

1. widen the mouth of the synthetic woman face, success. (logs/230602_080250)
2. widen the mouth of the real man face mentioned in the above comment, failure (logs/230602_081433)
3. drag up the eyebrow of the real man face, failure (logs/230602_083033)
   
   

In the optimization of 3rd case, the similarity of f1 and f2 is very large compared to the 1st case but it still can not succeed in moving the source points to the target.

I noticed the DragGAN paper uses the PTI for the GAN Inversion. In PTI paper, author claims that the editability of the inverted real image is poor and propose to fine-tune the generator to make inverted image resides in a more editable region. Therefore, can the problem be ameliorated by using PTI for the GAN Inversion?

[Zeqiang-Lai]

Wow, I haven't dive into the GAN inversion that original DragGAN paper uses before, but I think it would be a good idea to try more advanced inversion techniques.