4
$\begingroup$

I am using the following augmentations on dataset of size 9 GB:

datagen = ImageDataGenerator(
    featurewise_center=False,  # set input mean to 0 over the dataset
    samplewise_center=False,  # set each sample mean to 0
    featurewise_std_normalization=False,  # divide inputs by std of the dataset
    samplewise_std_normalization=False,  # divide each input by its std
    zca_whitening=True,  # apply ZCA whitening
    rotation_range=30,  # randomly rotate images in the range (degrees, 0 to 180)
    width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
    height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
    horizontal_flip=True,  # randomly flip images
    vertical_flip=True)  # randomly flip images
datagen.fit(a)

model.fit_generator(datagen.flow(a,b, batch_size=32),
                    steps_per_epoch=len(a) / 32, epochs=epochs, class_weight = sclass_weight, validation_data = [c, d],callbacks = [MetricsCheckpoint('logs')])

When the code comes to the datagen.fit, I get into memory error ( the code doesn't even go into training)

I have 50 gb ram and am training it on a K80 with a batch size of 32, so don't think that will be a problem.

It works fine when I comment all the augmentations.

Can someone tell me where I am going wrong?

$\endgroup$
  • $\begingroup$ You are running out of memory, because the computation are done on the GPU have a look at your house memory $\endgroup$ – Aditya May 31 '18 at 5:13
  • $\begingroup$ As I said earlier, gpu has 12 GB memory, and a batch size of 32 shouldn't be a problem right $\endgroup$ – Srihari May 31 '18 at 5:24
  • $\begingroup$ stackoverflow.com/q/49458905/6524169 this might help $\endgroup$ – Aditya May 31 '18 at 6:08
  • 1
    $\begingroup$ Are you using colab? due to using K80. $\endgroup$ – Media May 31 '18 at 7:13
  • $\begingroup$ @Media I don't think so as he has 50 gigs main mem $\endgroup$ – Aditya May 31 '18 at 7:40
0
$\begingroup$

It seems like the root cause for this is that zca_whitening is set to True. According to the answer from the keras-collaborator rragundez in this related Github issue, there is no work-around for it, except disabling ZCA whitening:

This is known problem with the ZCA method. There is no real immediate way around it. The problem is with the calculation of the sigma matrix which is a dot product.

The other alternative to disabling it would be to use images with a smaller resolution. But I suppose that's rarely a practical solution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.