What kind of Framework are you using? If you are using Keras, the newest Version of it supports multi-gpu How can i run a keras model on multi gpus
If you have an RNN network, using GPU wont give you much more performace. Most of the times CPU learning is faster in my experience.
Please provide more informations to your problem.
Update
If you have a more complex architecture in Tensorflow or Keras you can give each process Step in your Model a specific GPU/CPU which this part of the Model will be working on. You can do it with
with tf.device('/gpu:1'):
I think you just have to try it out. It depends on the Model you have( which you did not provide, so the complexity is not known) and on the volume of the data. Is your whole dataset in the RAM or do you generate each batch individually?
If you have several branches in your architecture and they are concatenating at a certain point. Which branch is the slowest?
First of all I would try to train your Model on the CPUs before doing anything else.
This post gives a quite good overview how you can use multi GPU.