Effect of batch size on dense networks?

Question

I have been doing DS for a couple of years now and have returned to "tinkering" a bit more with toy data sets and overall just honing my skills a bit.

I was recently playing with a very straightforward dataset that contains the history of 100 meter foot races (like in the Olympics). The data is not complex, just things like athlete height, age, etc.

I input the data into a dense network (Keras) and I was getting poor-to-fair results. However, this was all with a batch size of 1 but everything changed as soon as I worked with other batch sizes (2,4,8,16, etc). All my metrics went from poor to outstanding, even with just a batch size of 2.

Why is this? What is the layman's explanation for the effect batch sizes can have on a dense NN?

Answer 1

In general, the smaller the batch size the more noisy the gradient updates will be. That could lead to the network being unable to converge or take too long. There are also methods such as batchnorm which require a large enough batch size to effectively compute some statistics. On the other hand, the larger the batch size the larger the generalisation error.