2
$\begingroup$

I have been doing DS for a couple of years now and have returned to "tinkering" a bit more with toy data sets and overall just honing my skills a bit.

I was recently playing with a very straightforward dataset that contains the history of 100 meter foot races (like in the Olympics). The data is not complex, just things like athlete height, age, etc.

I input the data into a dense network (Keras) and I was getting poor-to-fair results. However, this was all with a batch size of 1 but everything changed as soon as I worked with other batch sizes (2,4,8,16, etc). All my metrics went from poor to outstanding, even with just a batch size of 2.

Why is this? What is the layman's explanation for the effect batch sizes can have on a dense NN?

$\endgroup$
1
1
$\begingroup$

In general, the smaller the batch size the more noisy the gradient updates will be. That could lead to the network being unable to converge or take too long. There are also methods such as batchnorm which require a large enough batch size to effectively compute some statistics. On the other hand, the larger the batch size the larger the generalisation error.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.