How we can program in the Keras library (or TensorFlow) to partition training on multiple GPUs? Let's say that you are in an Amazon ec2 instance that has 8 GPUs and you would like to use all of them to train faster, but your code is just for a single CPU or GPU.

  • 3
    $\begingroup$ have you checked the tensorflow doc ? $\endgroup$
    – n1tk
    Oct 19, 2017 at 2:25
  • $\begingroup$ @sb0709: I started reading this morning but I was wondering how to do it in keras $\endgroup$ Oct 19, 2017 at 2:31
  • 1
    $\begingroup$ don't know in keras but for tensorflow: tf will use GPU by default for computation even if is for CPU (if is present supported GPU). so you can just do a for loop: "for d in ['/gpu:1','/gpu:2', '/gpu:3' ... '/gpu:8',]:" and in the "tf.device(d)" should include all your instance GPU resources. So tf.device() will actually be used. $\endgroup$
    – n1tk
    Oct 19, 2017 at 2:34
  • $\begingroup$ Like this ?? for d in ['/gpu:1','/gpu:2', '/gpu:3' ... '/gpu:8',]: tf.device(d) and that is ? I will try like that :) $\endgroup$ Oct 19, 2017 at 2:36
  • 1
    $\begingroup$ as far I know yes, you can do any task on different device. $\endgroup$
    – n1tk
    Oct 19, 2017 at 2:38

4 Answers 4


From the Keras FAQs, below is copy-pasted code to enable 'data parallelism'. I.e. having each of your GPUs process a different subset of your data independently.

from keras.utils import multi_gpu_model

# Replicates `model` on 8 GPUs.
# This assumes that your machine has 8 available GPUs.
parallel_model = multi_gpu_model(model, gpus=8)

# This `fit` call will be distributed on 8 GPUs.
# Since the batch size is 256, each GPU will process 32 samples.
parallel_model.fit(x, y, epochs=20, batch_size=256)

Note that this appears to be valid only for the Tensorflow backend at the time of writing.

Update (Feb 2018):

Keras now accepts automatic gpu selection using multi_gpu_model, so you don't have to hardcode the number of gpus anymore. Details in this Pull Request. In other words, this enables code that looks like this:

    model = multi_gpu_model(model)

But to be more explicit, you can stick with something like:

parallel_model = multi_gpu_model(model, gpus=None)


To check if you really are utilizing all of your GPUs, specifically NVIDIA ones, you can monitor your usage in the terminal using:

watch -n0.5 nvidia-smi


  • 1
    $\begingroup$ Does the multi_gpu_model(model, gpus=None) work in the case where there is only 1 GPU? It would be cool if it automatically adapted to the number of GPUs available. $\endgroup$ Aug 30, 2018 at 6:25
  • $\begingroup$ Yes I think it works with 1 GPU, see github.com/keras-team/keras/pull/9226#issuecomment-361692460, but you might need to be careful that your code is adapted to run on a multi_gpu_model instead of a simple model. For most cases it probably wouldn't matter, but if you're going to do something like take the output of some intermediate layer, then you'll need to code accordingly. $\endgroup$
    – weiji14
    Sep 6, 2018 at 3:35
  • $\begingroup$ Do you have any references to multi gpu model differences? $\endgroup$ Sep 9, 2018 at 7:18
  • $\begingroup$ You mean something like github.com/rossumai/keras-multi-gpu/blob/master/blog/docs/…? $\endgroup$
    – weiji14
    Sep 10, 2018 at 4:17
  • $\begingroup$ That reference was great @weiji14. However I'm also interested in how this works for inference. Does keras somehow split batches equally or round robin schedule on available model replicas? $\endgroup$ Dec 10, 2018 at 6:00
  1. For TensorFlow:

TensorFlow Using GPUs

Here is the sample code on how is used, so for each task is specified the list with devices/device:

# Creates a graph.
c = []
for d in ['/gpu:2', '/gpu:3']:
  with tf.device(d):
    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3])
    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2])
    c.append(tf.matmul(a, b))
with tf.device('/cpu:0'):
  sum = tf.add_n(c)
# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
# Runs the op.

tf will use GPU by default for computation even if is for CPU (if is present supported GPU). so you can just do a for loop: "for d in ['/gpu:1','/gpu:2', '/gpu:3' ... '/gpu:8',]:" and in the "tf.device(d)" should include all your instance GPU resources. So tf.device() will actually be used.

Scaling Keras Model Training to Multiple GPUs

  1. Keras

For Keras by using Mxnet than args.num_gpus, where num_gpus is the list of your required GPUs.

def backend_agnostic_compile(model, loss, optimizer, metrics, args):
  if keras.backend._backend == 'mxnet':
      gpu_list = ["gpu(%d)" % i for i in range(args.num_gpus)]
          context = gpu_list)
      if args.num_gpus > 1:
          print("Warning: num_gpus > 1 but not using MxNet backend")
  1. horovod.tensorflow

On top of all Uber open sourced Horovod recently and I think is great:


import tensorflow as tf
import horovod.tensorflow as hvd

# Initialize Horovod

# Pin GPU to be used to process local rank (one GPU per process)
config = tf.ConfigProto()
config.gpu_options.visible_device_list = str(hvd.local_rank())

# Build model…
loss = …
opt = tf.train.AdagradOptimizer(0.01)

# Add Horovod Distributed Optimizer
opt = hvd.DistributedOptimizer(opt)

# Add hook to broadcast variables from rank 0 to all other processes during
# initialization.
hooks = [hvd.BroadcastGlobalVariablesHook(0)]

# Make training operation
train_op = opt.minimize(loss)

# The MonitoredTrainingSession takes care of session initialization,
# restoring from a checkpoint, saving to a checkpoint, and closing when done
# or an error occurs.
with tf.train.MonitoredTrainingSession(checkpoint_dir=“/tmp/train_logs”,
                                      hooks=hooks) as mon_sess:
 while not mon_sess.should_stop():
   # Perform synchronous training.

Basically, you can take example of the following example. All you need is specifying cpu and gpu consumption values after importing keras.

import keras

config = tf.ConfigProto( device_count = {'GPU': 1 , 'CPU': 56} )
sess = tf.Session(config=config) 

After then, you would fit the model.

model.fit(x_train, y_train, epochs=epochs, validation_data=(x_test, y_test))

Finally, you can decrease the consumption values not the work on upper limits.

  • $\begingroup$ Session seems to be in compat for Tensorflow 2.0, but not main function. $\endgroup$ Jan 6, 2020 at 20:13

Simple example for how we can access multiple GPUs with Horovd and Keras: Github code Keras MNIST Example with Horovod.

Plus, please go to the link for further info: Horovod with Keras


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.