0
$\begingroup$

Is there any work done on analyzing sequence of frames from a video using Deep Learning techniques?

By "analyzing" I mean like memorizing them in order to classify or predict something (e.g. by taking into account first 10 frames of a video the model can make some sort of conclusion).

$\endgroup$
2
$\begingroup$

In 2016 some people at MIT's CSAIL group "made an important new breakthrough in predictive vision, developing an algorithm that can anticipate interactions more accurately than ever before." They wrote an article, Teaching machines to predict the future.

They made a great video which shows their results. They trained an algorithm on YouTube videos and TV shows to predict when two individuals will shake hands, hug, kiss, or slap five.

enter image description here

The researchers, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba, published a paper at CVPR 2016 entitled Anticipating Visual Representations with Unlabeled Video.

Chia-Wen Cheng has a repository of an LSTM-based model implemented in TensorFlow.

For a more up to date list of papers on predicting actions or activities see this list.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for your help! I will accept this answer since it is more detailed :) $\endgroup$ – Stefan Radonjic Dec 26 '18 at 9:02
2
$\begingroup$

Yes there is. Actually using NNs on sequences is a big part of Deep Learning and one of the most powerful NNs come from this field of research. They are called Recurrent Neural Networks or RNNs. I would recommend you to read this article or watch this Stanford lecture, to learn more about them.

Especially a variant of RNNs, the so called LSTM that stands for Long Short Term Memory enables networks to memorize earlier inputs.

Based on your explanation I would suggest you have a many-to-one task where you input many frames and output one class. These types of tasks where you input images is most often solved using a combination of CNN and RNN. You could read this article to get more insights, to that technique.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.