0
$\begingroup$

I am working on a Deep Learning project and I am facing an issue with the size of the dataset. I want to make a pipeline for video dataset [Sequence Matters]. Because if I try the load the whole dataset then TensorFlow throughs an error which indirectly means out of memory. I read on the Official TensorFlow documentation about the tf.keras.preprocessing.image.ImageDataGenerator , tf.data and tf.data.Dataset for making pipeline for image dataset to load them in batches and avoid memory bottlenecks. This issue is that I want to do the same thing but with video dataset. As you know Videos consists of frames and the sequence of the frames matter a lot in recognition problems. I want to extract the frames from each video and load them in RAM in sequence to train my model, and I also want to achieve this in efficient manner (Loading 5 Videos samples at a given time to avoid full memory problem)

File Structure of the dataset is attached below:

enter image description here

Improve this question
$\endgroup$
5
  • $\begingroup$ Do you have a CPU or a GPU? GPUs could be much better in terms of performance, but more difficult for memory management. $\endgroup$
    – Nicolas Martin
    Commented Oct 21, 2022 at 7:48
  • $\begingroup$ I've a Nvidia Quadro P2000 with 5GB VRAM. And my systems RAM is 16 GB. The problem i am facing is that I'm not able to train my model. If i try to load the dataset then the Systems memory becomes full. I want an efficient way such that if i have let's say 15GB videos dataset. I want to load them whilst training. Not all at once. I hope you understand what i mean to say. If i have 500 Videos and the batch size is 33 then I don't need all the videos to be loaded in memory but just 32. I want to load only those videos we which my model is currently training on. Drop them and then load the next 32 $\endgroup$
    – Sharjeel M. Rajput
    Commented Nov 1, 2022 at 21:04
  • $\begingroup$ Can you help me with that? And also what if have i have 10GB dataset and i load 2GB dataset in RAM and train my model using model.fit and then save the model. Load the next 2GB dataset and retain the model with that new dataset by using model.fit again and do this for 5 times for all 10GB dataset. Would it work? Would my model will start the training from scratch if i apply model.fit again? In this way I won't have to deal with the memory full usage problem, i can simply train it on chunks of data until the model has seen the whole Dataset $\endgroup$
    – Sharjeel M. Rajput
    Commented Nov 1, 2022 at 21:08
  • $\begingroup$ I've answered below. Please let me know if it works. $\endgroup$
    – Nicolas Martin
    Commented Nov 1, 2022 at 22:21
  • $\begingroup$ Alright, I'll read it and let you know. $\endgroup$
    – Sharjeel M. Rajput
    Commented Nov 4, 2022 at 22:26

1 Answer 1

Reset to default
0
$\begingroup$

If your VRAM is limited, you should apply mini-batches as described in those articles:

https://blog.paperspace.com/how-to-maximize-gpu-utilization-by-finding-the-right-batch-size/

https://towardsdatascience.com/how-to-break-gpu-memory-boundaries-even-with-large-batch-sizes-7a9c27a400ce

If your mini-batches are quite small, maybe you would have to increase iterations in order to get good results.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you very much. I'll try this approach and will post my finding here in comments section. $\endgroup$
    – Sharjeel M. Rajput
    Commented Nov 4, 2022 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.