0
$\begingroup$

The dataset has 3-minute 30fps video conversations (no audio) of 150 extroverted and 150 introverted individuals. The goal is to classify them as "introverts" or "extroverts" based on their facial features.

I have extracted 35-dim facial action unit features using open-face for each frame. Then took an average of features from 5400 frames (for each video) for each sample and used an SVM classifier, resulting in 70% ACC.

  • What are other statistical measures to represent the entire data, instead of the average feature?
  • What are some other deep learning-based approaches for extracting/representing entire time series of facial features from videos and classifying binary personality types ("introvert" or "extrovert")?
$\endgroup$

1 Answer 1

0
$\begingroup$

You can use open-face to yield a 35-dim feature vector for each frame, as you said, and then treat each video as a sequence of features of shape (num_frames, 35) such that the entire dataset is of shape (300, num_frames, 35).

Then you can train a recurrent neural network classifier, like a LSTM that takes in a sequence of num_frames 35-dim vectors and yields a single number: the binary class.

Alternatively, I think you can replace the feature extraction by open-face with an end-to-end model by using 3D convolutional layers, and then an LSTM, with a final dense layer.

$\endgroup$
2
  • $\begingroup$ Tried that approach but got poor results. 3min@30fps = 5400 frames and LSTMs cannot handle capturing important features from longer sequences. There are most frames common between the 2 classes where the facial expression is neutral. $\endgroup$ Oct 3, 2023 at 15:58
  • $\begingroup$ @TheBiometricsGuy I see, what about using a transformer instead? Otherwise, you could try a sort of "hierarchical" approach in which you train a LSTM on T frames so having a total of 5400/T LSTMs, which, in turn, are aggregated by another LSTM or Dense layer: it's an idea, I don't know if it may work at all. $\endgroup$ Oct 4, 2023 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.