Why do I get an OOM error although my model is not that large?

Question

I am a newbie in GPU based training and deep learning models. I am running cDCGAN (Conditional DCGAN) in TensorFlow on my 2 Nvidia GTX 1080 GPUs. My data set consists of around 320,000 images with size 64*64 and 2,350 class labels. If I set my batch size 32 or larger I get an OOM error like below. So I am using 10 batch size for now.

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,64,64,2351] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[Node: discriminator/concat = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_41, _arg_Placeholder_3_0_3/_43, discriminator/concat/axis)]]
Caused by op 'discriminator/concat', defined at:
  File "cdcgan.py", line 221, in <module>
    D_real, D_real_logits = discriminator(x, y_fill, isTrain)
  File "cdcgan.py", line 48, in discriminator
    cat1 = tf.concat([x, y_fill], 3)

The training is very slow which I understand is down to the batch size (correct me if I am wrong). If I do help -n 1 nvidia-smi, I get the following output:

The GPU:0 is mainly used, as the Volatile GPU-Util gives me around 0%-65% whereas GPU:1 is always 0%-3% max. Performance for GPU:0 is always in P2 whereas GPU:1 is mostly P8 or sometimes P2. I have the following questions.

1. Why I am getting an OOM error on the large batch size although my dataset and model are not that big?

2. How can I utilize both GPUs equally in TensorFlow so that the performance is fast? (From the above error, it looks like GPU:0 gets full immediately whereas GPU:1 is not fully utilized. It is my understanding only).

Model details are as follows:

Generator:

I have 4 layers (fully connected, UpSampling2d-conv2d, UpSampling2d-conv2d, conv2d).
W1 is of the shape [X+Y, 1616128] i.e. (2450, 32768), w2 [3, 3, 128, 64], w3 [3, 3, 64, 32], w4 [[3, 3, 32, 1]] respectively

Discriminator

It has five layers (conv2d, conv2d, conv2d, conv2d, fully connected).
w1 [5, 5, X+Y, 64] i.e. (5, 5, 2351, 64), w2 [3, 3, 64, 64], w3 [3, 3, 64, 128], w4 [2, 2, 128, 256], [1616256, 1] respectively.

Session Configuration I am also allocating memory in advance via

gpu_options = tf.GPUOptions(allow_growth=True)
session = tf.InteractiveSession(config=tf.ConfigProto(gpu_options=gpu_options))

Answer 1

1. Why I am getting OOM error on the large batch size although my dataset and model are not that big?

Yes, the batch size is probably the reason.

Also, another the reason is that you don't use the second GPU at all (otherwise, both GPUs will split the batch - computation - and you could use larger batches).

2. How can I utilize both GPU's equally in Tensorflow so that the performance is fast? (From the above error, it looks like GPU:0 gets full immediately whereas GPU:1 is not fully utilized. it's my understanding only)

By default, Tensorflow occupies all available GPUs (that's way you see it with nvidia-smi - you have a process 34589 which took both GPUs), but, unless you specify in the code to actually use multi GPUs, it will use only one by default.

Here is an official TF docs about how to use multi GPUs: https://www.tensorflow.org/guide/using_gpu#using_multiple_gpus https://www.tensorflow.org/guide/gpu#using_multiple_gpus

Here is some tutorial for multi-gpu usage with more examples: https://jhui.github.io/2017/03/07/TensorFlow-GPU/

Answer 2

Blockquote 1) Why I am getting OOM error on the large batch size although my dataset and model are not that big?
Yes, the batch size is probably the reason.
Also, another the reason is that you don't use the second GPU at all (otherwise, both GPUs will split the batch - computation - and you could use larger batches).

No, the size of the batch is not the reason of yout OOM error if you use image $64\times 64$ and batch of $32$ the memory cost on GPU is:

$32\cdot 32\cdot 3$(size of image if in color)$\cdot 32$(size of batch)$\cdot 32$(bits for a float32) bits $ \approx 3MB$

The problem is certainly due to the size of your neural network. A GAN need to load the discriminator and the generator on the gpu when he train the generator.

You should calculate the size of both the discriminator and generator and add them to see if your GPU can load it

Answer 3

Coming here as this is a top google result for this issue, and reducing the batch size problem didn't help in my case. Here's my advice:

• If you are having this problem during training, my suggestion is to create a data generator. I posted an example in this SO answer.

• If you are having this problem during inference, the following might help:

  with tf.device("cpu:0"):
    prediction = model.predict(...)
  

  The difference between CPU and GPU inference time is not that high, and we'll have way more memory available using CPU.