41
$\begingroup$

I've seen discussions about the 'overhead' of a GPU, and that for 'small' networks, it may actually be faster to train on a CPU (or network of CPUs) than a GPU.

What is meant by 'small'?

For example, would a single-layer MLP with 100 hidden units be 'small'?

Does our definition of 'small' change for recurrent architectures?

Are there any other criteria that should be considered when deciding whether to train on CPU or GPU?

EDIT 1:

I just found a blog post (possibly outdated? It's from 2014):

"...Most network card[s] only work with memory that is registered with the CPU and so the GPU to GPU transfer between two nodes would be like this: GPU 1 to CPU 1 to Network Card 1 to Network Card 2 to CPU 2 to GPU 2. What this means is, if one chooses a slow network card then there might be no speedups over a single computer. Even with fast network cards, if the cluster is large, one does not even get speedups from GPUs when compared to CPUs as the GPUs just work too fast for the network cards to keep up with them.

This is the reason why many big companies like Google and Microsoft are using CPU rather than GPU clusters to train their big neural networks. "

So at some point, according to this post, it could have been faster to use CPUs. Is this still the case?

EDIT 2: Yes, that blog post may very well be outdated because:

Now it seems that GPUs within a node are connected via PCIe bus, so communication can happen at about 6GiB/s. (For example: https://www.youtube.com/watch?v=el1iSlP1uOs, about 35 minutes in). The speaker implies that this is faster than going from GPU1 to CPU to GPU2. It would mean the network card is no longer the bottleneck.

$\endgroup$
1
  • $\begingroup$ That guy with his blog post brings up good points. I did not understand all of his justifications. However, the fact that Google, Facebook, Twitter, and all the leading deep learning groups in academia run their codes primarily on GPUs suggests that it is a good idea. Although biased: nvidia.com/content/events/geoInt2015/LBrown_DL.pdf $\endgroup$
    – JahKnows
    May 30, 2017 at 17:05

3 Answers 3

35
$\begingroup$

Unlike some of the other answers, I would highly advice against always training on GPUs without any second thought. This is driven by the usage of deep learning methods on images and texts, where the data is very rich (e.g. a lot of pixels = a lot of variables) and the model similarly has many millions of parameters. For other domains, this might not be the case.

What is meant by 'small'? For example, would a single-layer MLP with 100 hidden units be 'small'?

Yes, that is definitely very small by modern standards. Unless you have a GPU suited perfectly for training (e.g. NVIDIA 1080 or NVIDIA Titan), I wouldn't be surprised to find that your CPU was faster.

Note that the complexity of your neural network also depends on your number of input features, not just the number of units in your hidden layer. If your hidden layer has 100 units and each observation in your dataset has 4 input features, then your network is tiny (~400 parameters). If each observation instead has 1M input features as in some medical/biotech contexts, then your network is pretty big in terms of number of parameters. For the remainder of my answer I'm assuming you have quite few input features pr. observation.

One good example I've found of comparing CPU vs. GPU performance was when I trained a poker bot using reinforcement learning. For reinforcement learning you often don't want that many layers in your neural network and we found that we only needed a few layers with few parameters. Moreover, the number of input features was quite low. Initially I trained on a GPU (NVIDIA Titan), but it was taking a long time as reinforcement learning requires a lot of iterations. Luckily, I found that training on my CPU instead made my training go 10x as fast! This is just to say that CPU's can sometimes be better for training.

Are there any other criteria that should be considered when deciding whether to train on CPU or GPU?

It's important to note that while on a GPU you will always want to fill up the entire GPU memory by increasing your batch size, that is not the case on the CPU. On the CPU an increase in batch size will increase the time pr. batch. Therefore, if it's important for you to have a very large batch size (e.g. due to a very noisy signal), it can be beneficial to use a GPU. I haven't experienced this in practice though and normally small batch sizes are preferred.

$\endgroup$
4
  • $\begingroup$ Thank you @pir! Do you have any specific references where I can read more? $\endgroup$ May 31, 2017 at 19:49
  • $\begingroup$ You can easily find the number of parameters of e.g. VGG to compare and see that your network is tiny in comparison. $\endgroup$
    – pir
    May 31, 2017 at 20:54
  • 3
    $\begingroup$ I haven't seen a lot of CPU/GPU comparisons on tiny networks because that's not what the big companies and research labs are interested in. $\endgroup$
    – pir
    May 31, 2017 at 20:55
  • $\begingroup$ @StatsSorceress If you want to test it on your own, why not just setup a simple Keras MLP and test the performance on GPU vs. CPU? Also, see my updated answer wrt. the size of your network. $\endgroup$
    – pir
    Jun 7, 2017 at 0:29
5
$\begingroup$

The CPU is the manager of the branch, he can do a bit of everything, but he is not great at much except delegating tasks. However, the GPU is a dedicated mathematician hiding in your machine. If you are doing any math heavy processes then you should use your GPU. Always.

If you are using any popular programming language for machine learning such as python or MATLAB it is a one-liner of code to tell your computer that you want the operations to run on your GPU.

You should also make sure to use all the cores of your machine. This means making use of parallel computing. Especially for neural networks where operations can be done independently, this is going to increase your speed immensely.

$\endgroup$
4
  • 7
    $\begingroup$ I have found that sometimes the overhead of transferring data to and from the GPU completely wipes out the speed increase from the parallelism. It's not always a good idea to go to GPU. $\endgroup$ Jun 8, 2017 at 16:48
  • 1
    $\begingroup$ It depends on the complexity of your model. If you are training a simple K-NN then perhaps it is not worthwhile. However, if you are training any model that requires an inverse matrix or a neural network which needs many consequential matrix operations it is always a good idea to opt for the GPU. $\endgroup$
    – JahKnows
    Jun 8, 2017 at 16:54
  • 1
    $\begingroup$ @AdrianKeister I agree. That what I was trying to get at in my answer. For the network mentioned by OP that would likely be the bottleneck. $\endgroup$
    – pir
    Jun 8, 2017 at 20:27
  • 1
    $\begingroup$ 100 hidden units is faster on GPU using my machine. I would need a very small number of hidden units for the CPU to be faster. Moreover, I always tend to do my training in batches. In this case I doubt a CPU will be the bottleneck considering data that is dense enough. $\endgroup$
    – JahKnows
    Jun 8, 2017 at 20:30
3
$\begingroup$

I'll first reference some quotes from similar questions:

When it comes to matrix operations, you don't think twice, you always opt for GPUs. source

The parallel architecture in a GPU is well adapted for vector and matrix operations. source

So if you read through these questions, you'll see that they advise to use GPU regardless of the case; it will always provide some improvement.

The reason you may have read that 'small' networks should be trained with CPU, is because implementing GPU training for just a small network might take more time than simply training with CPU - that doesn't mean GPU will be slower.

A 100-hidden unit network is kind of small, i'd call it a small network relative to the big deep networks out there. Recurrent architectures (mostly) have more synapses thant feed forward networks, so a 100-hidden units RNN is 'bigger' than a 100-hidden unit FFN.

$\endgroup$
5
  • $\begingroup$ Isn't it true that if you have an MLP with a single hidden layer of 100 units, that has the same number of parameters as a standard RNN with 100 hidden units because of weight sharing? It has more 'synapses' - more 'activations' - but the same number of parameters, right? $\endgroup$ May 30, 2017 at 16:31
  • $\begingroup$ i'm not familiar with the term 'weight' sharing. It has the same amount of activations, but more connections so more parameters... $\endgroup$ May 31, 2017 at 16:51
  • $\begingroup$ Weight sharing means that the weight matrix from one hidden layer in the RNN to the next hidden layer is the same; it's the same 'U' matrix, replicated across time. Also, the weights from the input to the hidden layer are the same across time. $\endgroup$ May 31, 2017 at 18:43
  • $\begingroup$ @StatsSorceress i'm not familiar with working with matrices. Yes, the weight matrix from a hidden layer to the next is the same. However, there are more connections in total (because a layer can also get connected to the PREVIOUS layer). I'm not sure how I can explain, but a RNN will always have more parameters as there are more connected layers.. $\endgroup$ May 31, 2017 at 18:49
  • $\begingroup$ Yes, I understand that there are physically more parameters, but many of those parameters take the same value, which means the effective number of parameters in an MLP and an RNN with the same number of input dimensions and same number of hidden dimensions will be the same. $\endgroup$ May 31, 2017 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.