Training Neural Network using TensorFlow on Large Video Dataset for Human Activity Recognition

Question

I am working on a Human Activity Recognition on videos using deep learning and want to train my Neural Network (ConvLSTM & LRCN) on my custom dataset. The issue that I am facing is that my dataset is more than 50 GB and I don't have that much memory to load it into the memory directly(16 GB RAM and Nvidia P2000 with 5 GB VRAM). I read about the TensorFlow flow_from_directory which is used to effectively load the dataset, but my problem is that I would like to load multiple images for single label.

For example, a video might have $n$ number of frames in it and I extract the required frames from it for training my NN on it, for instance a 30 sec video of walking label might have 2000 frames and I want to get every 20th frames (20,40,60...2000).

Now, how do I deal with sequence of these frames while using TensorFlow flow_from_directory? I have 40 Videos of each label (40 Walking, 40 Running, 40 jogging and so on) how do I tell TensorFlow that these $n$ frames belong to this single video(such as walking label) and how do I send them at once to the model to train on it?

As far as I know, the default behavior of flow_from_directory is that if we have 4 folder (cat, dog, zebra and fish) it go to each folder and gets each photo from it and assign the label same as the label name. But in my case I have for instance 3 folders (Walking, Running and Jogging) and in each folder I have Nested folders (Walking Video 1, Walking Video 2 and so on) and in each folder there are n number of frames belonging to that video label. How do I deal with that?

Nested folders Structures

Bold represents a folder

Walking (Walking Video 1 (n frames belong to video 1), Walking Video 2 (n frames belong to video 2) and so on for all the walking videos)

Running (Running Video 1 (n frames belong to video 1), Running Video 2 (n frames belong to video 2) and so on for all the Running videos)

and so on for all the labels

Answer 1

At an early stage, you don't have to use all the data, but rather build a first model with a reasonable amount of data (ex: 4Gb) for 3 reasons: 1- The learning time is much shorter, so you can apply a fast improvement loop. 2- The necessary hardware is more affordable. 3- It is easily expandable to 50Gb.

In addition to that, you can apply some preprocessing techniques to compress the data, lower definintion, or select the most meaningful scenes with good differentiability.

The scenes should also be comparable and have the same frequency and the same frame, so that your NN can easily find patterns at the correct time.

The frame rate could be lower: existing models can have good results with 15 fps or less. You could start with 15 fps to see if the results are correct, and increase or decrease it until reaching an optimal result.

Finally, once you master a model with a smaller dataset, you can extend it to 50Gb defining smaller batches and using RAM or hard disk memory. Otherwise, you can buy a better hardware or rent a super GPU in a cloud (cf. GCloud, AWS or Paperspace) for an affordable price.