2
$\begingroup$

I built a fairly standard backpropagation algorithm and just the process of forward propagating through a 5 layer x 5 nodes network using a data set of 10,000 observations of 39 variables takes almost 5 minutes for one iteration.

Do neural networks typically take many hours to train using data sets this size? My initial data set was 10x as long, but I couldn't wait an hour just for one forward pass to be completed.

$\endgroup$
  • $\begingroup$ What framework are you using? Most frameworks (such as MXNet) has the option to use GPU and even multiple GPUs. Without such speed up training will take forever. $\endgroup$ – Guy Apr 10 '17 at 3:08
  • $\begingroup$ see ai.stackexchange.com/a/5730 $\endgroup$ – Bob Mar 25 at 18:03
1
$\begingroup$

This is quite standard for the training time. It depends on how much optimization you did on your code. The speed of your processing unit, it's often better to use a GPU as opposed to a CPU. GPUs do mathematical operations much faster. Also, you should use parallel computing when you can, in the case of NN you definitely can.

Training a machine learning algorithm only needs to be done once. Let it run all night and then you will be ready to do some pretty good predictions.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ I optimized and tinkered to get the code to run parallel and the compute time went down to 5 seconds. That is a suspiciously large difference in time. All the data looks clean though. $\endgroup$ – milkmotel Mar 10 '17 at 21:50
  • $\begingroup$ Yeah that's normal almost all machine learning algorithms are ideal for parallel programming. You literally save time in a linear way. I have 240 cores. So with parallel computing it takes me 1/240th of the time. Avoiding for loops is a very good idea as well. Use matrix operations instead. Like numpy. $\endgroup$ – JahKnows Mar 10 '17 at 22:09
  • $\begingroup$ Loops seem unavoidable when you have to go layer by layer during backpropagation, but other than that, yes. I only have 2 cores (4 processors) on my machine so things still take a while $\endgroup$ – milkmotel Mar 10 '17 at 22:25
  • $\begingroup$ Be careful with that last statement. There is no reason to believe that a NN only needs to be trained once. Maybe once per iteration but if you only train it once, you are missing out on all kinds of possibilities $\endgroup$ – I_Play_With_Data Sep 20 '18 at 15:59
1
$\begingroup$

Neural networks typically take longer to run as you increase the number of features or columns in your dataset and also when you increase the number of hidden layers. Frameworks like tensorflow or Theano enable you to run your neural networks code on GPU to especially take advantage of the parallel programming capabilities for large array multiplications typical of backpropagation algorithms. My code to train a ConvNet for the Dogs vs Cats problem from kaggle took 50 mins to train on 24000 images. You can take a look at my experiments on the CIFAR10 dataset here

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.