0
$\begingroup$

I have made a convolutional neural network from scratch in python to classify the MNIST handwritten digits (centralized). It is composed of a single convolutional network with 8 3x3 kernels, a 2x2 maxpool layer and a 10 node dense layer with softmax as the activation function. I am using cross entropy loss and SGD.

When I train the network on the whole training set for a single epoch with a batch size of 1, I get 95% accuracy. However, when I try with a larger batch size (16, 32, 128), the learning becomes very noisy and the end accuracy is anywhere between 47%-86%. Why is it that my network performs so much worse and noisier on mini-batches?

Improve this question
$\endgroup$
2
  • $\begingroup$ Assuming that the gradients are averaged when using mini-batches, have you made sure that the number of weight updates is the same? $\endgroup$
    – Oxbowerce
    Commented Mar 10, 2021 at 17:45
  • $\begingroup$ I'm not sure what you mean by "the number of weight updates is the same", but yes, i am averaging gradients, including weight gradients during updating. But i no longer have this problem with the network since i've changed some things. I dont know what the problem was though. $\endgroup$
    – dontloseyourgoalie
    Commented Mar 15, 2021 at 9:53

1 Answer 1

Reset to default
0
$\begingroup$

Your model is very small for a convnet. 1 conv layer, 1 maxpool and 1 fc is very shallow. Try adding more layers and batchnorm2d after each conv layer followed by relu. No pool layers.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This would likely result in better performance, but do you have any idea why it is when I use mini-batches that the performance worsens? Is it bad to use mini batches on a shallow network? $\endgroup$
    – dontloseyourgoalie
    Commented Feb 4, 2021 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.