1
$\begingroup$

I am a newbie in GPU based training and Deep learning models. I am running cDCGAN (Conditonal DCGAN) in tensorflow on my 2 Nvidia GTX 1080 GPU's. My data-set consists of around 32,0000 images with size 64*64 and 2350 class labels. If I set my batch size 32 or large I get OOM error like below. So I am using 10 batch size for now.

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,64,64,2351] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[Node: discriminator/concat = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_41, _arg_Placeholder_3_0_3/_43, discriminator/concat/axis)]]
Caused by op 'discriminator/concat', defined at:
  File "cdcgan.py", line 221, in <module>
    D_real, D_real_logits = discriminator(x, y_fill, isTrain)
  File "cdcgan.py", line 48, in discriminator
    cat1 = tf.concat([x, y_fill], 3)

The training is very slow which I understand is down to the batch size (correct me if I am wrong). If I do help -n 1 nvidia-smi, I get the following output.

enter image description here

The GPU:0 is mainly used, as the Volatile GPU-Util gives me around 0%-65% whereas GPU:1 is always 0%-3% max. Performance for GPU:0 is always in P2 whereas GPU:1 is mostly P8 or sometimes P2. I have the following questions.

1) Why I am getting OOM error on the large batch size although my dataset and model are not that big?

2) How can I utilize both GPU's equally in Tensorflow so that the performance is fast? (From the above error, it looks like GPU:0 gets full immediately whereas GPU:1 is not fully utilized. it's my understanding only)

Model Details are as follows

Generator:

I have 4 layers (fully connected, UpSampling2d-conv2d, UpSampling2d-conv2d, conv2d).
W1 is of the shape [X+Y, 16*16*128] i.e. (2450, 32768), w2 [3, 3, 128, 64], w3 [3, 3, 64, 32], w4 [[3, 3, 32, 1]] respectively

Discriminator

It has five layers (conv2d, conv2d, conv2d, conv2d, fully connected).
w1 [5, 5, X+Y, 64] i.e. (5, 5, 2351, 64), w2 [3, 3, 64, 64], w3 [3, 3, 64, 128], w4 [2, 2, 128, 256], [16*16*256, 1] respectively.

Session Configuration I am also allocating memory in advance via

gpu_options = tf.GPUOptions(allow_growth=True)
session = tf.InteractiveSession(config=tf.ConfigProto(gpu_options=gpu_options))
$\endgroup$
3
$\begingroup$

1) Why I am getting OOM error on the large batch size although my dataset and model are not that big?

Yes, the batch size is probably the reason.

Also, another the reason is that you don't use the second GPU at all (otherwise, both GPUs will split the batch - computation - and you could use larger batches).

2) How can I utilize both GPU's equally in Tensorflow so that the performance is fast? (From the above error, it looks like GPU:0 gets full immediately whereas GPU:1 is not fully utilized. it's my understanding only)

By default, Tensorflow occupies all available GPUs (that's way you see it with nvidia-smi - you have a process 34589 which took both GPUs), but, unless you specify in the code to actually use multi GPUs, it will use only one by default.

Here is an official TF docs about how to use multi GPUs: https://www.tensorflow.org/guide/using_gpu#using_multiple_gpus

Here is some tutorial for multi-gpu usage with more examples: https://jhui.github.io/2017/03/07/TensorFlow-GPU/

$\endgroup$
  • $\begingroup$ Thank you for this detail explanation I will definitely set up multiple GPUs manually. Just one last thing, what do you mean by "some extra preprocessing/postprocessing in the graph and so on ..." Can you please give me a little overview? $\endgroup$ – Ammar Ul Hassan Mar 14 at 15:00
  • $\begingroup$ Since that thing is little out of scope here, I updated the answer and removed that part. The "batch size" reason will be enough here. :) $\endgroup$ – Antonio Jurić Mar 14 at 15:28
1
$\begingroup$

Blockquote 1) Why I am getting OOM error on the large batch size although my dataset and model are not that big?
Yes, the batch size is probably the reason.
Also, another the reason is that you don't use the second GPU at all (otherwise, both GPUs will split the batch - computation - and you could use larger batches).

No, the size of the batch is not the reason of yout OOM error if you use image $64\times 64$ and batch of $32$ the memory cost on GPU is:

$32\cdot 32\cdot 3$(size of image if in color)$\cdot 32$(size of batch)$\cdot 32$(bits for a float32) bits $ \approx 3MB$

The problem is certainly due to the size of your neural network. A GAN need to load the discriminator and the generator on the gpu when he train the generator.

You should calculate the size of both the discriminator and generator and add them to see if your GPU can load it

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.