# Accelerate deep learning model training on several GPUs

I have a deep learning model that can be trained in one GPU, however, is very slow. Is there a way to accelerate the training by parallelizing it across several GPUs? How would be the training process?

I can use any framework (pytorch, tensorflow, etc). And I know they accept multiple GPU and how to do it. I am more interested in how to spread, conceptually, a single model between GPUs to gain performance. I do mostly NLP models with RNN but I am also interested in CNN.

What kind of Framework are you using? If you are using Keras, the newest Version of it supports multi-gpu How can i run a keras model on multi gpus

If you have an RNN network, using GPU wont give you much more performace. Most of the times CPU learning is faster in my experience.

Update

If you have a more complex architecture in Tensorflow or Keras you can give each process Step in your Model a specific GPU/CPU which this part of the Model will be working on. You can do it with

with tf.device('/gpu:1'):


I think you just have to try it out. It depends on the Model you have( which you did not provide, so the complexity is not known) and on the volume of the data. Is your whole dataset in the RAM or do you generate each batch individually? If you have several branches in your architecture and they are concatenating at a certain point. Which branch is the slowest?

First of all I would try to train your Model on the CPUs before doing anything else.

This post gives a quite good overview how you can use multi GPU.

• @Escachator Updated the answer. I think you just have to try it out. – Mimi Müller Dec 26 '17 at 11:01
• Thanks, I know how to distribute the model through different gpus but I cannot see the benefits in terms of speed up the training or don't know how to do it so that it's faster. – Escachator Dec 26 '17 at 12:04
• Than you have to provide Code and sample data so it can be tested what is faster... – Mimi Müller Dec 26 '17 at 12:33
• I am asking conceptually what is the best way of split it on gpu. By layers on GPUs? Or whole model on each GPU and update of parameters to the whole when new minimum is found? What are usual strategies... Thanks – Escachator Dec 26 '17 at 12:51

there are two ways to parallel your model in multi-gpus. one is data parallel, it replicates whole model in each gpu, and you should input different data into different gpus; the cost you have to pay is that you have to merge gradient into one before updating the weights, so this method is more fit for convolution. another is model parallel, you split model into multi-parts, and each gpu is assigned to execute different parts; in this way, you have to pass parts output to another parts, so fully connected layer suits better.