What I want to do
Input an image-sequence (videoframes) that is already cropped and aligned to only the face/hair region and photorealistically replace this Person, Person A (can be anyone, I also want this to work cross-gender), with the face/hair region of another pre-defined Person B. Then output the resulting image-sequence of Person B seemingly mimicking Person A's face pose and expressions.
The project should preferably be available open source and on github for everyone to use and improve. It would also be nice if I could do it in Python 3.6.x.
What I have
- A very large dataset of images with Person B's face from all different angles and with all kinds of different facial expressions, aligned and centered (suitable for generative adversarial network training)
Ideas on how to do this
1. Database Approach
- Extract a multitude of landmarks of every image in the dataset of Person B and save them in a table with the reference to the image.
- Extract a multitude of landmarks for each image in the sequence of Person A and per-image select the closest matches in the table of Person B, then use these matching images as replacements.
I can see this approach resulting in a very long and resource intensive calculation with frames that just can not find a good match and may seem off or just plain fake.
2. ML Approach
- Use a BEGAN or WGAN process that will be trained similar to the famous male->female converters and then interpolate
This technique will result in a very low resolution output. All in all from the results I have seen in different github projects, it's only photorealistic up to 125x125 pixels
Why I'm writing this question
To ask all of you what direction you would recommend me to take =)