1
$\begingroup$

I have 2 gpus on my local machines, but i'm not sure that the model I am training is using both of them (the speed has not changed much).

My code:

def get_model():
    base_model = ResNet50(weights='imagenet', input_shape=(image_size,image_size,3), include_top=False)
    #base_model.trainable = False

    model = models.Sequential()
    model.add(base_model)
    model.add(layers.GlobalAveragePooling2D())
    model.add(layers.Dense(1024, activation='relu'))
    model.add(layers.Dropout(0.5))
    model.add(layers.Dense(196, activation='softmax'))
    model.summary()

    model = multi_gpu_model(model,gpus=2)

    #optimizer = optimizers.SGD(lr=1e-4, decay=1e-6, momentum=0.9, nesterov=True)
    optimizer = optimizers.RMSprop(lr=0.0001)
    model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['acc'])

    return model

I just added the multi_gpu_model setting, but am not sure that this is enough. I checked nvidia-smi every 0.5 seconds, but seems like only one gpu is working. How do I make sure that it uses the full 2 gpus?

$\endgroup$
1
$\begingroup$

If you are using Linux and Nvidia GPUs, you can do the following in a terminal

nvidia-smi

Which will show you some stats about the GPUs available on your system.

You can run it automatically every 2 seconds like this to see how power/memory usage changes during training:

watch -n 2 nvidia-smi

If you see nothing or the nvidia-smi command fails, you likely don't have the correct drivers installed.

If theya re showing, but Keras/Tensorflow is not finding them, have a look at this thread for more checks for Tensorflow backend.

$\endgroup$
  • $\begingroup$ yes I'm using 2 GPU's but I want to know I'm using 2 GPU parallelly when I'm training the neural network $\endgroup$ – monk Jul 19 at 22:53
  • $\begingroup$ Watching the stats shown in the output of nvidia-smi will show you that. If both are being heavily utilised at the same time, they are being used in parallel. $\endgroup$ – n1k31t4 Jul 19 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.