I managed to create an emotion recognition system that uses dense optical flow on each entire frame. While the accuracy range is within 80-90% with cross-validation, I am aiming to improve the accuracy of the program.

There are four emotions: Neutral, happy, surprised, and angry. So far my classifier works pretty well, though it tends to over guess 'neutral' when the answer is 'happy' or 'surprised'. This tends to happen when the mouth is only slightly opened, but the subject is visible smiling or have mouth opened in shock while the classifier still thinks the mouth is closed.

Confusion Matrix for Dense Optical Flow:   
[[27 22  0  0]  
 [ 0 57  1  0]  
 [ 0 12 60  0]  
 [ 0  9  3 68]]  
Accuracy: 80-90% range

There is something I want to try in order to solve this though.

I have the ability to get the position of facial landmarks, though I don't know what I can do to turn this information into an effective additional feature I can use that would increase the accuracy. I was thinking of just simply getting the face landmark coordinates at the end of each video, but I feel like that it would not be the solution to differentiate between a closed mouth and slightly opened one (the difference in coordinate values will be small I think and am guessing that machine learning won't notice the difference).

I considered the possibility of just simply taking a still image of the subject's mouth and just analyzing that, but rejected it as it vulnerable to factors like lighting and people's appearance, and inconsistent matrix sizes. Plus I want my additional feature to take advantage of facial movement tracking.

I was wondering if there is a smart way to implement facial landmark tracking into a feature that would increase the accuracy of my classifier by dealing with my classifier's tendency to over predict the emotion 'neutral'. Any ways I can accomplish that?

  • $\begingroup$ what are your feature vectors right now? $\endgroup$ – lollercoaster Jul 9 '15 at 17:38
  • $\begingroup$ My feature vectors are composed of numerical values acquired by dense optical flow (from OpenCV). Basically what happens is that there is a 17 frame video, and each frame is composed of 136 points used in dense optical flow. For each frame, the 136 point vector constantly concatenates with each other until the 17th frame to form a single 2312 sized vector. If there were to be a 2nd video, there will be another 2312 sized vector in the 2nd row. $\endgroup$ – user3377126 Jul 10 '15 at 2:56
  • $\begingroup$ each "point" is and (x, y), yes? so your dimensionality is 2312*2 = 4624? what classification algos are you using now? $\endgroup$ – lollercoaster Jul 10 '15 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.