How to work with large amount of data overcoming RAM issues in python

I use numpy arrays to work with deep learning images. But as the data gets bigger, I'm facing issue with RAM even before training the model when using techniques like data augmentation.

Can someone suggest me how to work with large data for eg. 30GB of data in my system which has 16gb ram.

P.S. I'm worried about RAM during preprocessing and training, while i do batch processing with my GPU

• Try the Theano framework in python. It maximizes utilization of GPU. Feb 10 '18 at 12:24
• Try using AWS :). It's fairly cheap and you can scale machine size to huge amounts of RAM. You can process your images on an AWS instance and move them to your local disk. Then you can just load data in batches when training your model. Feb 10 '18 at 22:05
• @MaximilianP I have a really good GPU which I don't want to waste. The problem I'm facing is the preprocessing which I don't know how to do in batches. That's where I need help in Feb 20 '18 at 17:52

Dask is designed to manage these types of workloads. It provides an interface like NumPy, Pandas, or Python iterators for larger-than-memory operations. An example of using Dask with TensorFlow can be found here.

• Thank you, that's very helpful. I will try this out soon. Feb 20 '18 at 17:54
• too late to the party but a quick one, is dask a replacement for pandas or does it depend on pandas? Apr 28 '18 at 12:19

Do all image preparation and data augmentation during preprocessing and save the result as arrays of one or more samples (up to mini-batch size). Do not convert the arrays back to images. Read these prepared arrays with the wrapper that trains your model. I recommend numpy.save for its simplicity and transparency. Other options are discussed here: Stackoverflow - persisting numpy arrays .

• Thats where i face memory issues, when i convert the image into array, the array itself is 12 gb itself. When i do data augmentation, it exceeds my ram. I want to work with sometime like using array from the hard disk instead of ram. I did see flow from directory in keras but that works when the images are arranged as directory but my images are organized using a csv Feb 20 '18 at 17:50

During image preprocessing in Keras, you may run out of memory when doing zca_whitening, which involves taking the dot product of an image with itself. This depends on the size of individual images in your dataset, not on the total size of your dataset.

The memory required for zca_whitening will exceed 16GB for all but very small images, see here for an explanation.

To solve this you can set zca_whitening=False in ImageDataGenerator.