In the last video of his course on Convolutional Neural Networks, Andrew Ng was discussing using ConvNets on 3D input data. He mostly discussed cat scans, but also mentioned you could treat a movie as 3D data and run a ConvNet on it (where the z axis is time, i.e. the # frame of the movie).

I find this pretty fascinating. I'm just curious if anybody is aware of anybody doing something like this in practice, and what they accomplished.


The 2015 Thumos challenge was about action recognition in video collected from youtube - totalling 13k videos (430 hrs or 45 million frames) and 101 'action classes'. The video is of activities like people brushing their teeth, playing basketball, and surfing, and some video clips included multiple activities.

The video is untrimmed, so it's closer to a real-world setting than video edited down to exclusively a single action of interest. The winning and second place entries for action classification included CNNs in various guises (after a quick scan I couldn't find any entries that didn't).

Another use you might be interested in is video captioning (not classification): this paper incorporates CNNs to describe in words what is taking place in each video (examples from p9 of the article, I love the descriptions at bottom right): HRNE video descriptions

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.