I'm running a deep learning neural network that has been trained by a GPU. I now want to deploy this to multiple hosts for inference. The question is what are the conditions to decide whether I should use GPU's or CPUs for inference?
Adding more details from comments below.
I'm new to this so guidance is appreciated.
Memory: GPU is K80
Framework: Cuda and cuDNN
Data size per workloads: 20G
Computing nodes to consume: one per job, although would like to consider a scale option
Cost: I can afford a GPU option if the reasons make sense
Deployment: Running on own hosted bare metal servers, not in the cloud.
Right now I'm running on CPU simply because the application runs ok. But outside of that reason, I'm unsure why one would even consider GPU.