Can someone explain what batch size is doing in convolutional NNs?

Question

I've noticed that the performance of my models vary quite a bit as a function of the batch size, both in terms of the time to converge and (possibly) the amount of overfitting.I thought batch size was simply controlling the number of images sent to the gpu / cpu at any given time, but I can't match that up with this behavior. Can someone explain what batch size controls in the model? Why the variation in performance? Any good resources?

Sorry for the naive question. This is my first time using these sorts of models, and in the background reading I've done, I've still not fully grasped what this parameter is doing.

Answer 1

Batch size determines how often the weights are updated during training. The smaller the batch size, the more frequent the updates. On the other hand, when the batch size is small, updates are made without evaluating a large portion of the data at hand so at times, the gradient may be moving in a direction other than minimum if that particular batch is not a good representative of the entire dataset. So in a sense, a large batch size helps you get to a lower loss faster compared to a smaller one - it takes relatively larger steps with little noise. In practice, one rarely uses the entire dataset because it takes too much time per iteration. Of course for smaller dataset, this is not a problem.

Here you can find a good discussion of this issue by Andrew Ng.

Having said that, I also vaguely remember a paper discouraging from using too large batches (more than 1024). Smaller batches might wiggle the loss left and right too much before reaching a minimum, but they will get closer to it. And also some discussion about training with very large batches not performing as well on validation sets. So there is a trade-off and the optimal value should be picked by trial and error.

Answer 2

Batch size controls the number of updates that is applied to your model. Supposed you have $n$ images and you uses batch size of $b$ you can expect $\lfloor \frac{n}{b}\rfloor$ updates on your model. Increasing $b$ will obviously reduce the number of updates that is done and decreasing b will increase the number of updates.

One of the general consensus regarding batch size that is larger batch size leads to poorer generalization. This cross-validated link provides a good discussion and answer regarding this issue. This behaviour is unfavorable especially for large-scale industry uses when you have the resources and the number of data to train your model but you also do not want to sacrifice your performance as well. Another thing is gradient descent optimization is a sequential process so parallelization is out of question. Hence, people have been experimenting on large-scale/batch size training and one of the paper by facebook research gives a good insight how to train such model.

Answer 3

Here are some posts that I found helpful. Generally, the smaller the batch size the noisier the updates, so if you decrease the batch size you should probably decrease the learning rate, and train for more iterations.

I found the two links below helpful. One is a blog post from Ilya Sutskever, which has some practical advice at the bottom. Also, see the Caffe thread. They talk a bit about how to adjust the learning rate for smaller batch sizes.

http://yyue.blogspot.com/2015/01/a-brief-overview-of-deep-learning.html

https://github.com/BVLC/caffe/issues/430