Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have a supply of images of a specific cartoon character's face. I have hours of video. I would like to automatically find the sections of the video in which this cartoon character appears.
https://github.com/ageitgey/face_recognition doesn't seem to work very well on cartoon characters (it fails to recognize any face in my images).
What is the state of the art on this? Is there an open-source library or framework that is good at this?
You will most likely need to train a neural network to detect your particular cartoon characters. Although some parts of this are tedious, this type of task is well-documented and there are many user-friendly frameworks available. I would recommend reading the tensorflow object detection api tutorial.
The most difficult part will simply be collecting your data. You will need to sample video frames to train the neural network. Depending on how many characters are in your cartoon, the complexity and variability of the inputs, and network model that you choose, I suspect you will need to collect anywhere from 200-500 distinct samples for training.
After you collect your frames, you will need to annotate your data. "Annotation" is the process by which you manually draw bounding boxes around your characters so that the neural network knows what to look for. This process is described in further detail in the above link. Fortunately, you do not need to program the annotation tool yourself; the Tensorflow tutorial instructs you to install LabelImg, which provides a graphical interface for you to label your selected frames.
After creating your dataset, you can proceed to train your network using the Tensorflow instructions.
If you have a lot of video to inference, I would recommend sampling your frames in relatively large intervals such as 5 or even 10 (depending on the fps of the video). You can then inference 5 to 10 times faster with reasonable accuracy.
For example, suppose your network inferences frames $ x_t $ through $ x_{t+10} $ in 5 frame intervals. If both $ x_t $ and $ x_{t+5} $ have at least one bounding box of reasonable size, then frames between $ x_t $ and $ x_{t+5} $ most likely have characters as well. If frame $ x_{t+10} $ does not have a bounding box of reasonable size, then we might assume that the character(s) leave the frames at about $ x_{t+7} $ or $ x_{t+8} $. This method will let you control the inherent speed/accuracy trade-off.
