1
$\begingroup$

I use numpy arrays to work with deep learning images. But as the data gets bigger, I'm facing issue with RAM even before training the model when using techniques like data augmentation.

Can someone suggest me how to work with large data for eg. 30GB of data in my system which has 16gb ram.

P.S. I'm worried about RAM during preprocessing and training, while i do batch processing with my GPU

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Try the Theano framework in python. It maximizes utilization of GPU. $\endgroup$
    – user-116
    Commented Feb 10, 2018 at 12:24
  • $\begingroup$ Try using AWS :). It's fairly cheap and you can scale machine size to huge amounts of RAM. You can process your images on an AWS instance and move them to your local disk. Then you can just load data in batches when training your model. $\endgroup$
    – MaximilianP
    Commented Feb 10, 2018 at 22:05
  • 1
    $\begingroup$ @MaximilianP I have a really good GPU which I don't want to waste. The problem I'm facing is the preprocessing which I don't know how to do in batches. That's where I need help in $\endgroup$
    – Srihari
    Commented Feb 20, 2018 at 17:52

3 Answers 3

Reset to default
2
$\begingroup$

Dask is designed to manage these types of workloads. It provides an interface like NumPy, Pandas, or Python iterators for larger-than-memory operations. An example of using Dask with TensorFlow can be found here.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you, that's very helpful. I will try this out soon. $\endgroup$
    – Srihari
    Commented Feb 20, 2018 at 17:54
  • $\begingroup$ too late to the party but a quick one, is dask a replacement for pandas or does it depend on pandas? $\endgroup$
    – PirateApp
    Commented Apr 28, 2018 at 12:19
2
$\begingroup$

Do all image preparation and data augmentation during preprocessing and save the result as arrays of one or more samples (up to mini-batch size). Do not convert the arrays back to images. Read these prepared arrays with the wrapper that trains your model. I recommend numpy.save for its simplicity and transparency. Other options are discussed here: Stackoverflow - persisting numpy arrays .

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thats where i face memory issues, when i convert the image into array, the array itself is 12 gb itself. When i do data augmentation, it exceeds my ram. I want to work with sometime like using array from the hard disk instead of ram. I did see flow from directory in keras but that works when the images are arranged as directory but my images are organized using a csv $\endgroup$
    – Srihari
    Commented Feb 20, 2018 at 17:50
1
$\begingroup$

During image preprocessing in Keras, you may run out of memory when doing zca_whitening, which involves taking the dot product of an image with itself. This depends on the size of individual images in your dataset, not on the total size of your dataset.

The memory required for zca_whitening will exceed 16GB for all but very small images, see here for an explanation.

To solve this you can set zca_whitening=False in ImageDataGenerator.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.