23
$\begingroup$

I'm running a deep learning neural network that has been trained by a GPU. I now want to deploy this to multiple hosts for inference. The question is what are the conditions to decide whether I should use GPU's or CPUs for inference?

Adding more details from comments below.

I'm new to this so guidance is appreciated.

  • Memory: GPU is K80

  • Framework: Cuda and cuDNN

  • Data size per workloads: 20G

  • Computing nodes to consume: one per job, although would like to consider a scale option

  • Cost: I can afford a GPU option if the reasons make sense

  • Deployment: Running on own hosted bare metal servers, not in the cloud.

Right now I'm running on CPU simply because the application runs ok. But outside of that reason, I'm unsure why one would even consider GPU.

Improve this question
$\endgroup$
2
  • $\begingroup$ To advice on comparison between two potential approaches, it will be helpful for others to know some details of your task. For example, what is the size of your data, what is the memory capacity of your GPU, the number of computing nodes you plan on using and perhaps also what map-reduce framework you have in mind. $\endgroup$
    – Dynamic Stardust
    Sep 26, 2017 at 23:05
  • $\begingroup$ @DynamicStardust ...and the cost. Cheap EC2 CPU nodes or expensive ECW GPU nodes? This is way too vague a question. $\endgroup$
    – Spacedman
    Sep 27, 2017 at 6:42

3 Answers 3

Reset to default
18
$\begingroup$

It is true that for training a lot of the parallalization can be exploited by the GPU's, resulting in much faster training. For Inference, this parallalization can be way less, however CNN's will still get an advantage from this resulting in faster inference. Now you just have to ask yourself: is faster inference important? Do I want this extra dependencies (a good GPU, the right files installed etc)?

If speed is not an issue, go for CPU. However note that GPU's can make it an order of magnitude faster in my experience.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Interesting point you raise, why would CNN benefit from parallel processing during inference? $\endgroup$
    – Dan
    Sep 28, 2017 at 15:10
  • 1
    $\begingroup$ With training the parallel calculations might be obvious: you have multiple inputs, that each have to be feed-forwarded, totally independent of each other. For a convolution, the kernel is multiplicated over multiple input 'patches'. These can be done parallel too. That's what makes CNN so powerful: not only do they need less parameters to train on, it's also more parallelized, hence making GPU's so powerfull. I don't know what you work on, but I work on segmentation (pixelwise classification) of images and when using GPU's for inference I get a huge speed improvement (>x10). $\endgroup$
    – Laurens Meeus
    Sep 28, 2017 at 15:18
  • $\begingroup$ @LaurensMeeus I'm also new to this spectrum and am doing cost analysis of cloud VMs. Would I still use GPU for training if im only training text analysis and not images? $\endgroup$
    – J.S. Orris
    Jul 31, 2019 at 15:47
  • 2
    $\begingroup$ @Squ1rr3lz I'm 95% you should. Every form of parallel computing should get an advantage on GPU's. I'm not an expert in this field, but given that text analysis is also with convolutional layers (be it 1D instead of 2D), this is already one reason it could/should be faster. If possible, just try for yourself what influence enabling/disabling the GPU has. $\endgroup$
    – Laurens Meeus
    Aug 6, 2019 at 11:47
  • $\begingroup$ To speed up inference on GPU, the inputs must be in batch. However, many times the input is a single data point, i.e. a piece of tweet. In such a case, how GPU can speed up inference? $\endgroup$
    – marlon
    Nov 30, 2020 at 4:43
8
$\begingroup$

Running inference on a GPU instead of CPU will give you close to the same speedup as it does on training, less a little to memory overhead.

However, as you said, the application runs okay on CPU. If you get to the point where inference speed is a bottleneck in the application, upgrading to a GPU will alleviate that bottleneck.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Totally agree. Still don't get why the CPU post is topped. $\endgroup$
    – Laurens Meeus
    Oct 13, 2017 at 9:07
  • 1
    $\begingroup$ GeForce GTX Titan X delivers between 5.3 and 6.7 times higher performance than the 16-core Intel Xeon E5 CPU -- that's much lower than the speedup achieved during training -- from NVidia's own blog: devblogs.nvidia.com/… $\endgroup$
    – seanhalle
    May 4, 2018 at 20:47
5
$\begingroup$

You'd only use GPU for training because deep learning requires massive calculation to arrive at an optimal solution. However, you don't need GPU machines for deployment.

Let's take Apple's new iPhone X as an example. The new iPhone X has an advanced machine learning algorithm for facical detection. Apple employees must have a cluster of machines for training and validation. But your iPhone X doesn't need a GPU for just running the model.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thanks for that explanation. What I don't understand then is why is Nvidia promoting the use of GPU's as there best solution for inference if CPU can do it fine. Also why are Google also pushing TPU for inference if they do it all with CPU? Does it have anything to do with parallel computing? $\endgroup$
    – Dan
    Sep 27, 2017 at 9:20
  • 3
    $\begingroup$ @SmallChess didn't IPhone X have some advanced TPU-like processor in it purely for inference? extremetech.com/mobile/… $\endgroup$
    – Laurens Meeus
    Sep 28, 2017 at 8:36
  • 3
    $\begingroup$ @SmallChess But why can't it be? Maybe I just don't completely understand. Won't you still get some improvement in speed? $\endgroup$
    – Laurens Meeus
    Sep 28, 2017 at 8:38
  • 3
    $\begingroup$ Honestly, NVidia has a fantastic marketing department, and it is in their best interest to promote using NVidia GPUs for inference. It is in their best interest, not yours. $\endgroup$
    – seanhalle
    May 4, 2018 at 20:35
  • 1
    $\begingroup$ From NVidia's blog: "258 vs. 242 images/second" for NVIDIA Tegra X1 vs i7 6700K CPU when performing inference: devblogs.nvidia.com/… $\endgroup$
    – seanhalle
    May 4, 2018 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.