# Memory error on using data generator in keras

I am using the following augmentations on dataset of size 9 GB:

datagen = ImageDataGenerator(
featurewise_center=False,  # set input mean to 0 over the dataset
samplewise_center=False,  # set each sample mean to 0
featurewise_std_normalization=False,  # divide inputs by std of the dataset
samplewise_std_normalization=False,  # divide each input by its std
zca_whitening=True,  # apply ZCA whitening
rotation_range=30,  # randomly rotate images in the range (degrees, 0 to 180)
width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
horizontal_flip=True,  # randomly flip images
vertical_flip=True)  # randomly flip images
datagen.fit(a)

model.fit_generator(datagen.flow(a,b, batch_size=32),
steps_per_epoch=len(a) / 32, epochs=epochs, class_weight = sclass_weight, validation_data = [c, d],callbacks = [MetricsCheckpoint('logs')])


When the code comes to the datagen.fit, I get into memory error ( the code doesn't even go into training)

I have 50 gb ram and am training it on a K80 with a batch size of 32, so don't think that will be a problem.

It works fine when I comment all the augmentations.

Can someone tell me where I am going wrong?

• You are running out of memory, because the computation are done on the GPU have a look at your house memory – Aditya May 31 '18 at 5:13
• As I said earlier, gpu has 12 GB memory, and a batch size of 32 shouldn't be a problem right – Srihari May 31 '18 at 5:24
• stackoverflow.com/q/49458905/6524169 this might help – Aditya May 31 '18 at 6:08
• Are you using colab? due to using K80. – Media May 31 '18 at 7:13
• @Media I don't think so as he has 50 gigs main mem – Aditya May 31 '18 at 7:40

It seems like the root cause for this is that zca_whitening is set to True. According to the answer from the keras-collaborator rragundez in this related Github issue, there is no work-around for it, except disabling ZCA whitening: