Training Deep Nets on an Ordinary Laptop

Question

Would it be possible for a an amateur who is interested in getting some "hands-on" experience in desining and training deep neural networks, to use an ordinary laptop for that purpose (no GPU), or is it hopeless to get good results in reasonable time without a powerful computer/cluster/GPU?

To be more specific, the laptop's CPU is an Intel Core i7 5500U fith generation, with 8GB RAM.

Now, since I haven't specified what problems I would like to work on, I'll frame my questions in a different way: which deep architectures would you recommend that I try to implement with my hardware, such that the following goal is achieved: Acquiring intuition and knowledge about how and when to use techniques that were introduced in the past 10 years and were essential to the uprising of deep nets (such as understanding of initialisations, drop-out, rmsprop, just to name a few).

I read about these techniques, but of course without trying them out myself I wouldn't know exactly how and when to implement these in an effective way. On the other hand, I'm afraid that if I try using a PC which isn't strong enough, then my own learning rate will be so slow that it would be meaningless to say that I've acquired any better understanding. And if I try using these techniques on shallow nets, maybe I wouldn't be building the right intuition.

I imagine the process of (my) learning as follows: I implement a neural net, let it practice for up to several hours, see what I've got, and repeat the process. If I do this once or twice a day, I would be happy if after, say, 6 months I will have gained practical knowledge which is comparable to what a professional in the field should know.

Answer 1

Yes, a laptop will work just fine for getting acquainted with some deep learning projects:

You can pick a smallish deep learning problem and gain some tractable insight using a laptop so give it a try.

The Theano project has a set of tutorials on digit recognition that I've played with and moded on a laptop.

Tensorflow also has a set of tutorials.

I let some of the longer runs go overnight, but nothing was intractable.

You might also consider availing yourself of AWS or one of the other cloud services. For 20-30 dollars you can perform some of the bigger calculations in the cloud on some sort of elastic computing node. The secondary advantage is that you can also list AWS or other cloud services as skill on your resume also :-)

Hope this helps!

Answer 2

[...] is it hopeless to get good results in reasonable time without a powerful computer/cluster/GPU?

It's not hopeless and you can, without doubt, gain lots of relevant experience with deep learning using the computer spec you mentioned. It will come down to your neural network architecture (number of layers and neurons), size of the dataset (number of inputs), nature of the data (inherent patterns), and implementation. And although you may need to limit yourself with those regards it won't prevent you from acquiring intuition and knowledge you're referring to. You'll easily experience problems of overfitting, influence of regularization, effects of pre-training, impact of different neuron types and architectures to name a few.

I'll give you a more concrete example. I've implemented a couple of deep learning algorithms (all CPU-based) in Julia and run them on a MacBook Air (similar to your spec). The code was not terribly optimised as neurons and layers were represented by actual data structures rather than a single giant matrix. So further performance improvements were possible.

For a fully-connected network of 56x300x300x300x1 (56 inputs and approx 200k connections) and 250 training examples I was able to get 5k back propagation passes within a day. Often that was enough to overfit the data or perfectly fit the training set (but this will depend on your dataset and other aforementioned factors). If the data has strong patterns and less than 10k examples you often won't need that many iterations. It's not uncommon that few hundreds of pre-training and refinement iterations lead to good results. So yes, your laptop is good enough and you could run meaningful experiments that take several hours.

[...] which deep architectures would you recommend that I try to implement with my hardware, such that the following goal is achieved: Acquiring intuition and knowledge about how and when to use techniques that were introduced in the past 10 years and were essential to the uprising of deep nets.

I'd suggest to pick smaller datasets with strong patterns. And I'd recommend to look into pre-training techniques such as auto-encoders because they often require fewer iterations to reach better results. Start with back propagation and build from there, try different architectures, neuron types, use regularization, auto-encoders, dropout, ...

Also make sure to pick a performant language or library for your experiments.