28
$\begingroup$

When training neural networks, one hyperparameter is the size of a minibatch. Common choices are 32, 64 and 128 elements per mini batch.

Are there any rules / guidelines how big a mini-batch should be? Any publications which investigates the effect on the training?

$\endgroup$
  • $\begingroup$ Other than fitting in memory? $\endgroup$ – Ehsan M. Kermani Apr 17 '17 at 16:28
  • $\begingroup$ Yes. For example, is there any publication with says "the bigger the batch size, the better" (as long as it fits in memory)? $\endgroup$ – Martin Thoma Apr 17 '17 at 16:29
  • $\begingroup$ @EhsanM.Kermani I think it does matter. I made a couple of runs on CIFAR-100 and I get different results depending on the batch size (with early stopping so that overfitting is hopefully not a problem) $\endgroup$ – Martin Thoma Apr 17 '17 at 16:45
  • 3
    $\begingroup$ Bigger computes faster (is efficient), smaller converges faster, generalizes better; cf. Efficient Mini-batch Training for Stochastic Optimization and this RNN study. There is a sweet spot that you find empirically for your problem. $\endgroup$ – Emre Apr 17 '17 at 17:00
  • 2
    $\begingroup$ This most insightful paper by Blei et al just came out: Stochastic Gradient Descent as Approximate Bayesian Inference $\endgroup$ – Emre Apr 17 '17 at 21:04
29
$\begingroup$

In On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima there are a couple of intersting statements:

It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize [...]

large-batch methods tend to converge to sharp minimizers of the training and testing functions—and as is well known, sharp minima lead to poorer generalization. n. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation.

From my masters thesis: Hence the choice of the mini-batch size influences:

  • Training time until convergence: There seems to be a sweet spot. If the batch size is very small (e.g. 8), this time goes up. If the batch size is huge, it is also higher than the minimum.
  • Training time per epoch: Bigger computes faster (is efficient)
  • Resulting model quality: The lower the better due to better generalization (?)

It is important to note hyper-parameter interactions: Batch size may interact with other hyper-parameters, most notably learning rate. In some experiments this interaction may make it hard to isolate the effect of batch size alone on model quality. Another strong interaction is with early stopping for regularisation.

See also

$\endgroup$
  • $\begingroup$ @NeilSlater Do you want to add your comment to my (now community wiki) answer? $\endgroup$ – Martin Thoma Apr 17 '17 at 18:17
  • $\begingroup$ I like the answer as a general one. Moreover I would appreciate to have a number about what are very small, huge and mini-batch in a specific example. $\endgroup$ – So S Apr 19 '17 at 10:32
  • $\begingroup$ @SoS mini-batch is just a term. The "mini" does not refer to a specific size, but it only means that there is more than 1 example and less than the total training set. I consider "very small" to be <= 8 (I've just edited the answer). I also measured an extreme (more than 5x) increase in wall-clock training time for this. Normal is something like 64 or 128. I'm not too sure what "huge" is; I think this might depend on the hardware. $\endgroup$ – Martin Thoma Apr 19 '17 at 11:20
  • $\begingroup$ This answer asks more questions than it answers. Where is this sweet spot (maybe a graph would help)? How does it interact with learning rate and early stopping? $\endgroup$ – xjcl Sep 3 '19 at 1:13
  • $\begingroup$ The answer depends on the network and the dataset. Hence it doesn't make sense to give specific numbers and hence a graph would not help. About interactions with other hyperparameters: I don't know for sure. Try it and publish your results :-) $\endgroup$ – Martin Thoma Sep 3 '19 at 5:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.