3
$\begingroup$

I was trying the Keras CNN Stater Code on Ubuntu 16.04, from the below link: https://www.hackerearth.com/challenge/competitive/deep-learning-3/machine-learning/predict-the-energy-used-612632a9/#c144537

I get “MemoryError:” for

X_train = np.array(train_img, np.float32) / 255.

Any idea, what should I be doing?

$\endgroup$
3
  • $\begingroup$ See if this SO answer helps. $\endgroup$ May 7 '18 at 13:22
  • 1
    $\begingroup$ That link requires registration so I'd suggest you describe the best you can the problem with code snippets included. In addition you should post the specs of your computer in case it's just a simple "not enough RAM error" which appears to be the case. $\endgroup$
    – wacax
    May 7 '18 at 22:39
  • $\begingroup$ @wacax Below are the code lines just below the above error giving line: from tqdm import tqdm def read_img(img_path): img = cv2.imread(img_path) img = cv2.resize(img, (128, 128)) return img train_img = [] for img_path in tqdm(train.Image_name.values): train_img.append(read_img(TRAIN_PATH + img_path)) Any idea how to know how much space that variable is taking ? And how to decide min Gb of RAM required to solve such problems? $\endgroup$
    – Nands
    May 8 '18 at 15:07
3
$\begingroup$

MemoryError is exactly what it means, you have run out of memory in your RAM for your code to execute.

When this error occurs it is likely because you have loaded the entire data into memory. For large datasets you will want to use batch processing. Instead of loading your entire dataset into memory you should keep your data in your hard drive and access it in batches. If you are using Keras there is a helper class with a very efficient implementation of batch processing. Take a look at this blog post. This is a good starting point for avoiding MemoryError.


As a short term fix you can train your model using a subset of the data available to you and discard the rest. Doing this really is a shame however.

$\endgroup$
3
  • $\begingroup$ If you have additional questions we can help you write the DataGenerator class to suit your data. $\endgroup$
    – JahKnows
    May 7 '18 at 13:30
  • $\begingroup$ Thanks @JahKnows. I will go through the blog post, see if/how I can implement it in my case, and let you know. $\endgroup$
    – Nands
    May 8 '18 at 14:58
  • $\begingroup$ hi @JahKnows I have a similar question, could you please help: stackoverflow.com/questions/59222403/… $\endgroup$
    – Franva
    Dec 7 '19 at 2:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.