1
$\begingroup$

So I'm a newcomer to the world of neural networks, and I've been getting a little familiar with the field and have started playing with my own networks.

I'm using Lasagne, and I'm finding that the training of my NN is taking an infeasible amount of time. Maybe the problem I'm looking at is simply biting more than my computer can chew; I just don't know, because I'm not familiar enough with how these things go.

I'm trying to train a 1D convolutional NN, which has two 1D convolution layers, and two max-pooling layers, and two fully connected layers. All in all, the NN has around 95,000 parameters (weights + biases). I'm running this on an Amazon EC2 GPU instance. What I'm finding is that during training, it's taking 0.3 sec/sample.

By contrast, when I run the Lasagne MNIST CNN example, it's taking about 0.0003 sec/sample, 1000 times faster! Now to be fair, my input samples are 18000x3 dimensional, and the MNIST samples are 28x28 dimensional; but the difference there is a factor of 68. And the MNIST CNN actually has more parameters than mine, with 160,000 or so parameters.

Does this indicate anything wrong with what I'm doing?

Improve this question
$\endgroup$
1
  • $\begingroup$ Size of the convolutions will have a large impact. Typically for MNIST-solving CNNs, each convolution patch is 5x5 or 7x7, and is convolved over 28x28 space (e.g. around 400 times per patch, if done naively). What are your figures for your 1d convolutions? In 1D, you could easily have a patch 1x100 (larger than MNIST) and running it fully over 18000 length is going to be required almost 18000 times - I can easily see a factor of 1000 creeping in there. You might want to look at FFT style convolutions, not sure if Lasagne supports them (for small-to-mid sized patches, they are not efficient) $\endgroup$ – Neil Slater Oct 8 '15 at 10:36
1
$\begingroup$

Found the answer

I found out why it was going slowly, after doing some profiling and reading this:

It turns out that if you do a convolution with a stride != 1, then it doesn't use the GPU for that operation (or perhaps the gradient of that operation), and makes things run much slower. I was only using a non-1 stride to save execution time anyway, but turns out I'm better off getting rid of the stride.

Also note that in your convoution layer, you have to completely get rid of stride=X. Just having stride=1 is not sufficient.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.