My understanding is that GPUs are more efficient for running neural nets, but someone recently suggested to me that GPUs are only needed for the training phase. Once trained, it's actually more efficient to run them on CPUs.
Is this true?
This depends on many factors, such as the neural network architecture (CNNs tend to be better optimized than RNN on GPU) as well as how many test samples you give as input to the neural network (GPUs can be even faster when given a batch of samples instead of a single sample).
As an example, here is a benchmark comparing CPU with GPU on different CNN-based architectures. The forward pass is much slower on a CPU in that case: