I'm running a deep learning neural network that has been trained by a GPU. I now want to deploy this to multiple hosts for inference. The question is what are the conditions to decide whether I should use GPU's or CPUs for inference?

Adding more details from comments below.

I'm new to this so guidance is appreciated.

  • Memory: GPU is K80

  • Framework: Cuda and cuDNN

  • Data size per workloads: 20G

  • Computing nodes to consume: one per job, although would like to consider a scale option

  • Cost: I can afford a GPU option if the reasons make sense

  • Deployment: Running on own hosted bare metal servers, not in the cloud.

Right now I'm running on CPU simply because the application runs ok. But outside of that reason, I'm unsure why one would even consider GPU.

  • $\begingroup$ To advice on comparison between two potential approaches, it will be helpful for others to know some details of your task. For example, what is the size of your data, what is the memory capacity of your GPU, the number of computing nodes you plan on using and perhaps also what map-reduce framework you have in mind. $\endgroup$ – Dynamic Stardust Sep 26 '17 at 23:05
  • $\begingroup$ @DynamicStardust ...and the cost. Cheap EC2 CPU nodes or expensive ECW GPU nodes? This is way too vague a question. $\endgroup$ – Spacedman Sep 27 '17 at 6:42

@Dan @SmallChess, I don't completely agree. It is true that for training a lot of the parallalization can be exploited by the GPU's, resulting in much faster training. For Inference, this parallalization can be way less, however CNN's will still get an advantage from this resulting in faster inference. Now you just have to ask yourself: is faster inference important? Do I want this extra dependencies (a good GPU, the right files installed etc)?

If speed is not an issue, go for CPU. However note that GPU's can make it an order of magnitude faster in my experience.

  • $\begingroup$ Interesting point you raise, why would CNN benefit from parallel processing during inference? $\endgroup$ – Dan Sep 28 '17 at 15:10
  • 1
    $\begingroup$ With training the parallel calculations might be obvious: you have multiple inputs, that each have to be feed-forwarded, totally independent of each other. For a convolution, the kernel is multiplicated over multiple input 'patches'. These can be done parallel too. That's what makes CNN so powerful: not only do they need less parameters to train on, it's also more parallelized, hence making GPU's so powerfull. I don't know what you work on, but I work on segmentation (pixelwise classification) of images and when using GPU's for inference I get a huge speed improvement (>x10). $\endgroup$ – Laurens Meeus Sep 28 '17 at 15:18
  • $\begingroup$ @LaurensMeeus I'm also new to this spectrum and am doing cost analysis of cloud VMs. Would I still use GPU for training if im only training text analysis and not images? $\endgroup$ – Squ1rr3lz Jul 31 '19 at 15:47
  • 1
    $\begingroup$ @Squ1rr3lz I'm 95% you should. Every form of parallel computing should get an advantage on GPU's. I'm not an expert in this field, but given that text analysis is also with convolutional layers (be it 1D instead of 2D), this is already one reason it could/should be faster. If possible, just try for yourself what influence enabling/disabling the GPU has. $\endgroup$ – Laurens Meeus Aug 6 '19 at 11:47

Running inference on a GPU instead of CPU will give you close to the same speedup as it does on training, less a little to memory overhead.

However, as you said, the application runs okay on CPU. If you get to the point where inference speed is a bottleneck in the application, upgrading to a GPU will alleviate that bottleneck.

  • $\begingroup$ Totally agree. Still don't get why the CPU post is topped. $\endgroup$ – Laurens Meeus Oct 13 '17 at 9:07
  • 1
    $\begingroup$ GeForce GTX Titan X delivers between 5.3 and 6.7 times higher performance than the 16-core Intel Xeon E5 CPU -- that's much lower than the speedup achieved during training -- from NVidia's own blog: devblogs.nvidia.com/… $\endgroup$ – seanhalle May 4 '18 at 20:47

You'd only use GPU for training because deep learning requires massive calculation to arrive at an optimal solution. However, you don't need GPU machines for deployment.

Let's take Apple's new iPhone X as an example. The new iPhone X has an advanced machine learning algorithm for facical detection. Apple employees must have a cluster of machines for training and validation. But your iPhone X doesn't need a GPU for just running the model.

  • $\begingroup$ Thanks for that explanation. What I don't understand then is why is Nvidia promoting the use of GPU's as there best solution for inference if CPU can do it fine. Also why are Google also pushing TPU for inference if they do it all with CPU? Does it have anything to do with parallel computing? $\endgroup$ – Dan Sep 27 '17 at 9:20
  • 3
    $\begingroup$ @SmallChess didn't IPhone X have some advanced TPU-like processor in it purely for inference? extremetech.com/mobile/… $\endgroup$ – Laurens Meeus Sep 28 '17 at 8:36
  • 3
    $\begingroup$ @SmallChess But why can't it be? Maybe I just don't completely understand. Won't you still get some improvement in speed? $\endgroup$ – Laurens Meeus Sep 28 '17 at 8:38
  • 2
    $\begingroup$ Honestly, NVidia has a fantastic marketing department, and it is in their best interest to promote using NVidia GPUs for inference. It is in their best interest, not yours. $\endgroup$ – seanhalle May 4 '18 at 20:35
  • 1
    $\begingroup$ From NVidia's blog: "258 vs. 242 images/second" for NVIDIA Tegra X1 vs i7 6700K CPU when performing inference: devblogs.nvidia.com/… $\endgroup$ – seanhalle May 4 '18 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.