0
$\begingroup$

I have made a convolutional neural network from scratch in python to classify the MNIST handwritten digits (centralized). It is composed of a single convolutional network with 8 3x3 kernels, a 2x2 maxpool layer and a 10 node dense layer with softmax as the activation function. I am using cross entropy loss and SGD.

When I train the network on the whole training set for a single epoch with a batch size of 1, I get 95% accuracy. However, when I try with a larger batch size (16, 32, 128), the learning becomes very noisy and the end accuracy is anywhere between 47%-86%. Why is it that my network performs so much worse and noisier on mini-batches?

$\endgroup$
2
  • $\begingroup$ Assuming that the gradients are averaged when using mini-batches, have you made sure that the number of weight updates is the same? $\endgroup$ – Oxbowerce Mar 10 at 17:45
  • $\begingroup$ I'm not sure what you mean by "the number of weight updates is the same", but yes, i am averaging gradients, including weight gradients during updating. But i no longer have this problem with the network since i've changed some things. I dont know what the problem was though. $\endgroup$ – dontloseyourgoalie Mar 15 at 9:53
0
$\begingroup$

Your model is very small for a convnet. 1 conv layer, 1 maxpool and 1 fc is very shallow. Try adding more layers and batchnorm2d after each conv layer followed by relu. No pool layers.

$\endgroup$
1
  • $\begingroup$ This would likely result in better performance, but do you have any idea why it is when I use mini-batches that the performance worsens? Is it bad to use mini batches on a shallow network? $\endgroup$ – dontloseyourgoalie Feb 4 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.