I have a large training set of ~300GB (which is a subset of an even larger dataset ~15TB).

I am trying train a Convnet with Keras (Tensorflow backend) to do something similar to semantic segmentation.

I couldn't find any valuable resources to handle such large data. Any suggestions for best practices for such humungous data is appreciated.



You don't need to load the whole dataset into memory at once. The only data you need in memory are the samples in a single training batch. Use the fit_generator method rather than fit to pass in an iterator that feeds samples to your model from disk rather than loading all of that data at once. Here's a tutorial that discusses this more.

| improve this answer | |
  • $\begingroup$ Thanks, a lot! The solution works well for me even though its a bit slow since multiple reads from the disk have to be made during every iteration. $\endgroup$ – Arun Aniyan Mar 8 '18 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.