2
$\begingroup$

What I want to do

Input an image-sequence (videoframes) that is already cropped and aligned to only the face/hair region and photorealistically replace this Person, Person A (can be anyone, I also want this to work cross-gender), with the face/hair region of another pre-defined Person B. Then output the resulting image-sequence of Person B seemingly mimicking Person A's face pose and expressions.

The project should preferably be available open source and on github for everyone to use and improve. It would also be nice if I could do it in Python 3.6.x.

What I have

  • A very large dataset of images with Person B's face from all different angles and with all kinds of different facial expressions, aligned and centered (suitable for generative adversarial network training)

Ideas on how to do this

1. Database Approach

Process

  • Extract a multitude of landmarks of every image in the dataset of Person B and save them in a table with the reference to the image.
  • Extract a multitude of landmarks for each image in the sequence of Person A and per-image select the closest matches in the table of Person B, then use these matching images as replacements.

Concerns

I can see this approach resulting in a very long and resource intensive calculation with frames that just can not find a good match and may seem off or just plain fake.

2. ML Approach

Process

  • Use a BEGAN or WGAN process that will be trained similar to the famous male->female converters and then interpolate

Concerns

This technique will result in a very low resolution output. All in all from the results I have seen in different github projects, it's only photorealistic up to 125x125 pixels

Why I'm writing this question

To ask all of you what direction you would recommend me to take =)

$\endgroup$
  • $\begingroup$ Excluding hair, something like arxiv.org/pdf/1704.06729.pdf might work. Adding hair, I'm not sure if there is anything to be found, because hair is very hard to model and account for occlusions etc in the source image. And as you have noted, technologies like GAN, VAE, style transfer might be promising, but cannot make such a sophisticated and targeted edit. In film and TV editing, such a change is still a skilled and labour intensive task. $\endgroup$ – Neil Slater Jul 20 '17 at 7:19
  • $\begingroup$ thank you @NeilSlater !! looks very good, although it looks just a tad to supervised for my desired purpose, I will give it a try =) $\endgroup$ – Rainbow Unicorn Jul 23 '17 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.