My understanding is that GPUs are more efficient for running neural nets, but someone recently suggested to me that GPUs are only needed for the training phase. Once trained, it's actually more efficient to run them on CPUs.
Is this true?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
This depends on many factors, such as the neural network architecture (CNNs tend to be better optimized than RNN on GPU) as well as how many test samples you give as input to the neural network (GPUs can be even faster when given a batch of samples instead of a single sample).
As an example, here is a benchmark comparing CPU with GPU on different CNN-based architectures. The forward pass is much slower on a CPU in that case: