I am a newbie in GPU based training and deep learning models. I am running cDCGAN (Conditional DCGAN) in TensorFlow on my 2 Nvidia GTX 1080 GPUs. My data set consists of around 320,000 images with size 64*64 and 2,350 class labels. If I set my batch size 32 or larger I get an OOM error like below. So I am using 10 batch size for now.
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,64,64,2351] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: discriminator/concat = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_41, _arg_Placeholder_3_0_3/_43, discriminator/concat/axis)]] Caused by op 'discriminator/concat', defined at: File "cdcgan.py", line 221, in <module> D_real, D_real_logits = discriminator(x, y_fill, isTrain) File "cdcgan.py", line 48, in discriminator cat1 = tf.concat([x, y_fill], 3)
The training is very slow which I understand is down to the batch size (correct me if I am wrong). If I do
help -n 1 nvidia-smi, I get the following output:
GPU:0 is mainly used, as the Volatile GPU-Util gives me around 0%-65% whereas
GPU:1 is always 0%-3% max. Performance for
GPU:0 is always in P2 whereas
GPU:1 is mostly P8 or sometimes P2. I have the following questions.
Why I am getting an OOM error on the large batch size although my dataset and model are not that big?
How can I utilize both GPUs equally in TensorFlow so that the performance is fast? (From the above error, it looks like GPU:0 gets full immediately whereas GPU:1 is not fully utilized. It is my understanding only).
Model details are as follows:
I have 4 layers (fully connected, UpSampling2d-conv2d, UpSampling2d-conv2d, conv2d).
W1 is of the shape [X+Y, 1616128] i.e. (2450, 32768), w2 [3, 3, 128, 64], w3 [3, 3, 64, 32], w4 [[3, 3, 32, 1]] respectively
It has five layers (conv2d, conv2d, conv2d, conv2d, fully connected).
w1 [5, 5, X+Y, 64] i.e. (5, 5, 2351, 64), w2 [3, 3, 64, 64], w3 [3, 3, 64, 128], w4 [2, 2, 128, 256], [1616256, 1] respectively.
Session Configuration I am also allocating memory in advance via
gpu_options = tf.GPUOptions(allow_growth=True) session = tf.InteractiveSession(config=tf.ConfigProto(gpu_options=gpu_options))