3
$\begingroup$

I'm trying to run a deep model using GPU and seems Keras running the validation against the whole validation data set in one batch instead of validating in many batches and that's causing out of memory problem

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[160000,64,64,1] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:GatherV2]

I did not have this problem when I was running on CPU, it's just happening when I'm running on GPU, my fit code looks like this

history =model.fit(patches_imgs_train,patches_masks_train, batch_size=8, epochs=10,
  shuffle=True, verbose=1, validation_split=0.2) 

When I delete the validation parameter from the fit method the code works, but I need the validation.

$\endgroup$
1
$\begingroup$

There may be two causes to your problem:

  • In validation the network needs more memory.
  • There is another problem not directly related to this.

In this Keras issue you can find a discussion of a very similar problem. Basically, you can try:

  • Reducing the batch size.
  • If you are using Tensorboad, try disabling it or setting its batch_size parameter.
$\endgroup$
3
  • $\begingroup$ As far as I know, the validation and training data use the same batch size but seems it's not the case here and the batch_size=8 is not applied on the validation, there is no param for validation batch size, would you advise me how I can compute the validation loss batch by batch. $\endgroup$ – Yassir Jun 26 '20 at 9:39
  • $\begingroup$ Sorry, I understood something different from your question. This answer is not correct, let me fix it. $\endgroup$ – noe Jun 26 '20 at 9:55
  • $\begingroup$ Thanks @ncasas, I saw this thread in GitHub before I think there is a bug when it comes to Keras GPU, reducing the batch size won't have a real impact as the validation is not using it, I'm not using the Tensorboard, I'm really confused and not sure what to do and I really need the validation, is there a way maybe to use GPU for training and CPU for validation maybe. $\endgroup$ – Yassir Jun 26 '20 at 10:19
0
$\begingroup$

So I could consider what is happening as a bug in Keras implementation, looks like it's trying to load the whole data set to the memory for splitting it into validation and training sets and it's not related to batch size, after trying many ways to go around it I found the best way to approach it is splitting the data using sklearn train_test_split instead of splitting it down in the fitting method using validation_split param.

x_train, x_v, y_train, y_v = train_test_split(x,y,test_size = 0.2,train_size =0.8)

history = model.fit(x_train,y_train,
              batch_size=16,
              epochs=5,
              shuffle=True,
              verbose=2,
              validation_data=(x_v, y_v))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.