I have a training dataset of images common images, there are more than 5K images in this dataset. But I have less memory in Google colab- RAM-12GB.

I need to train all the images but due to less memory, I can't.

What are the possible ways to train all the images with less memory?

I have an idea, but don't know it is an optimal solution, which is

I split the dataset into 5 sets[each set contains 1000 images] and train the 1 set of dataset. Then, using the model file, I will train the 2 nd set of dataset, then again load the updated model file, I will train the 3rd set of dataset, and continues...

If I followed this steps, then it means that I trained all the images in the dataset?

Thanks for your help

  • $\begingroup$ Welcome to DataScienceSE. You could first try to do an ablation study with a subset of the data in order to estimate how many instances the model really needs. Maybe the performance stops increasing after 1000 images. $\endgroup$
    – Erwan
    Nov 14, 2021 at 23:31
  • $\begingroup$ @Erwan, could you please share any study material for the ablation study links which you think the best, thanks $\endgroup$
    – Vishak Raj
    Nov 15, 2021 at 4:46
  • $\begingroup$ See this answer for instance. $\endgroup$
    – Erwan
    Nov 15, 2021 at 10:04


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.