0
$\begingroup$

Ran my CNN on a SageMaker notebook and it started training, but I had to restart the kernel due to AWS disconnecting. However when I tried to rerun my code, I received an OOM error, and it never started training again. I tried:

  • Restarting the kernel
  • Restarted the AWS machine But the error still persisted. I find this strange due to the fact it ran before.

ResourceExhaustedError: OOM when allocating tensor with shape[262145,25600] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:RandomUniform]

$\endgroup$

2 Answers 2

1
$\begingroup$

This might be due to 2 possible reasons:

  1. First time you ran the model, it got loaded into the GPU memory. When you disconnected and connected again, the was still loaded into the memory. When you again ran the notebook, it tried to load the model but could not as there was already a model loaded. If this is the case what you could do is kill all python PID's and try again.
  2. It may happen sometimes that when the batch size is too big, the initial few epochs may run properly but subsequent epochs may not run due to OOM error. In this case, you can decrease the batch size and try again.
$\endgroup$
0
$\begingroup$

You can decrease the batch size of the data in model.fit.for example if you have set batch size 32 you can change it to 16 ot 8

$\endgroup$
1
  • $\begingroup$ I shouldn't need to do this though because it was running before $\endgroup$ Commented Aug 20, 2021 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.