Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have a deep learning model that can be trained in one GPU, however, is very slow. Is there a way to accelerate the training by parallelizing it across several GPUs? How would be the training process?
I can use any framework (pytorch, tensorflow, etc). And I know they accept multiple GPU and how to do it. I am more interested in how to spread, conceptually, a single model between GPUs to gain performance. I do mostly NLP models with RNN but I am also interested in CNN.
What kind of Framework are you using? If you are using Keras, the newest Version of it supports multi-gpu How can i run a keras model on multi gpus
If you have an RNN network, using GPU wont give you much more performace. Most of the times CPU learning is faster in my experience.
Please provide more informations to your problem.
Update
If you have a more complex architecture in Tensorflow or Keras you can give each process Step in your Model a specific GPU/CPU which this part of the Model will be working on. You can do it with
with tf.device('/gpu:1'):
I think you just have to try it out. It depends on the Model you have( which you did not provide, so the complexity is not known) and on the volume of the data. Is your whole dataset in the RAM or do you generate each batch individually? If you have several branches in your architecture and they are concatenating at a certain point. Which branch is the slowest?
First of all I would try to train your Model on the CPUs before doing anything else.
This post gives a quite good overview how you can use multi GPU.
there are two ways to parallel your model in multi-gpus. one is data parallel, it replicates whole model in each gpu, and you should input different data into different gpus; the cost you have to pay is that you have to merge gradient into one before updating the weights, so this method is more fit for convolution. another is model parallel, you split model into multi-parts, and each gpu is assigned to execute different parts; in this way, you have to pass parts output to another parts, so fully connected layer suits better.
