My understanding is that GPUs are more efficient for running neural nets, but someone recently suggested GPUs are only needed for the training phase and that once trained, it's actually more efficient to run them on CPUs.

Is this true?


That would depend entirely on what software platform you're running on, not on any particular characteristic of neural nets or their constituent objects. For example, Theano leverages GPUs because they're optimized for matrix math, which neural net weights and activations are often expressed as. On the other hand, they can also be expressed as sets, which makes them ideal for set-based languages like SQL; I prefer this approach, since sets are more flexible (you can, for example, operate easily on ragged dimensional sets, unlike with matrices) and SQL is a somewhat portable standard that makes it easier to think about neural net layers conceptually. As my screen name suggests, I use SQL Server, which is one of many database server platforms that can do some serious number-crunching on very large neural nets, all without the use of any GPU power. Long ago I also programmed neural nets in VB.Net, VB 6 and C# that efficiently made use of arrays, without leveraging GPUs at all.

I think the choice may come down to a combination of your existing skill set, what architectures you have access to, your portability requirements, code maintenance issues and all that, not to any explicit linkage between specific hardware components and the inner workings of neural nets. There may be slight advantages to modeling ragged matrices in SQL or to the matrix-crunching power of GPUs, but I don't think there's much practical difference in most scenarios, nor can I think of a way of testing such a comparison without getting into an apples vs. oranges conundrum. That being said, within a specific platform, it may be true that "GPUs are only needed for the training phase" and "once trained, it's actually more efficient to run them on CPUs." That I can't comment on, since I still with SQL and the .Net stack and avoid Theano etc. altogether (though I respect their capabilities). I hope that helps.

| improve this answer | |
  • $\begingroup$ Very interesting, never heard of anyone using SQL Server to run neural nets! Is actually very common? But the training is definitely faster on GPUs? $\endgroup$ – Crashalot Nov 6 '16 at 5:13
  • $\begingroup$ I don't think it would be faster at all on GPUs than CPUs if the software accessing them is structured completely differently . Database servers like Oracle/SQL Server have all kinds of special memory management, indexing, execution plan and other enhancements that let them leverage CPU power in really optimal ways, which makes comparisons to tools like Theano that leverage GPU power like apples vs. oranges. It's going to also be highly dependent on how the SQL or other code is written is as well, not so much on the hardware it runs on top of. $\endgroup$ – SQLServerSteve Nov 6 '16 at 6:23
  • $\begingroup$ At least on the VB.Net, VB 6, SQL Server stack I'm familiar with, there really isn't much difference between running activation functions in recall algorithms and updating connection weights in learning algorithms. The objects used and the math operations performed on them aren't much different. Whether or not that holds for a different platform where one can switch between CPU and GPU calculations I cannot say for sure, but I don't see why there'd be much difference (unless the platform creators baked some kind of dependency into them). $\endgroup$ – SQLServerSteve Nov 6 '16 at 6:26
  • $\begingroup$ Thanks for sharing! Is using SQL Server or database servers a common approach to running/training neural nets? $\endgroup$ – Crashalot Nov 6 '16 at 7:27
  • $\begingroup$ I don't think it's common at all - I think it will be in the future though since there are certain advantages to database serves that tools like R, Theano etc. can't leverage. SQL Server has also had Microsoft-engineered feedforward nets with backprop included in its data mining components for ~12 years, but the user community's small. Cortana can access SQL Server & has neural nets but there are many limitations. I prefer being able to code my own nets from scratch in SQL and VB, rather than be limited by some other organization's design decisions. I hope that helps. $\endgroup$ – SQLServerSteve Nov 6 '16 at 9:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.