When training neural networks, one hyperparameter is the size of a minibatch. Common choices are 32, 64, and 128 elements per mini batch.

Are there any rules/guidelines on how big a mini-batch should be? Or any publications which investigate the effect on the training?

  • $\begingroup$ Other than fitting in memory? $\endgroup$ Apr 17, 2017 at 16:28
  • $\begingroup$ Yes. For example, is there any publication with says "the bigger the batch size, the better" (as long as it fits in memory)? $\endgroup$ Apr 17, 2017 at 16:29
  • $\begingroup$ @EhsanM.Kermani I think it does matter. I made a couple of runs on CIFAR-100 and I get different results depending on the batch size (with early stopping so that overfitting is hopefully not a problem) $\endgroup$ Apr 17, 2017 at 16:45
  • 3
    $\begingroup$ Bigger computes faster (is efficient), smaller converges faster, generalizes better; cf. Efficient Mini-batch Training for Stochastic Optimization and this RNN study. There is a sweet spot that you find empirically for your problem. $\endgroup$
    – Emre
    Apr 17, 2017 at 17:00
  • 3
    $\begingroup$ This most insightful paper by Blei et al just came out: Stochastic Gradient Descent as Approximate Bayesian Inference $\endgroup$
    – Emre
    Apr 17, 2017 at 21:04

2 Answers 2


In On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima there are a couple of intersting statements:

It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize [...]

large-batch methods tend to converge to sharp minimizers of the training and testing functions—and as is well known, sharp minima lead to poorer generalization. n. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation.

From my masters thesis: Hence the choice of the mini-batch size influences:

  • Training time until convergence: There seems to be a sweet spot. If the batch size is very small (e.g. 8), this time goes up. If the batch size is huge, it is also higher than the minimum.
  • Training time per epoch: Bigger computes faster (is efficient)
  • Resulting model quality: The lower the better due to better generalization (?)

It is important to note hyper-parameter interactions: Batch size may interact with other hyper-parameters, most notably learning rate. In some experiments this interaction may make it hard to isolate the effect of batch size alone on model quality. Another strong interaction is with early stopping for regularisation.

See also

  • $\begingroup$ @NeilSlater Do you want to add your comment to my (now community wiki) answer? $\endgroup$ Apr 17, 2017 at 18:17
  • 1
    $\begingroup$ I like the answer as a general one. Moreover I would appreciate to have a number about what are very small, huge and mini-batch in a specific example. $\endgroup$
    – So S
    Apr 19, 2017 at 10:32
  • $\begingroup$ @SoS mini-batch is just a term. The "mini" does not refer to a specific size, but it only means that there is more than 1 example and less than the total training set. I consider "very small" to be <= 8 (I've just edited the answer). I also measured an extreme (more than 5x) increase in wall-clock training time for this. Normal is something like 64 or 128. I'm not too sure what "huge" is; I think this might depend on the hardware. $\endgroup$ Apr 19, 2017 at 11:20
  • $\begingroup$ This answer asks more questions than it answers. Where is this sweet spot (maybe a graph would help)? How does it interact with learning rate and early stopping? $\endgroup$
    – xjcl
    Sep 3, 2019 at 1:13
  • $\begingroup$ The answer depends on the network and the dataset. Hence it doesn't make sense to give specific numbers and hence a graph would not help. About interactions with other hyperparameters: I don't know for sure. Try it and publish your results :-) $\endgroup$ Sep 3, 2019 at 5:22

Based upon Andrew Ng's Deep Learning Specialisation Course 2, here're a few things to be kept in mind:

  1. Use mini-batch gradient descent if you have a large training set. Else for a small training set, use batch gradient descent.
  2. Mini-batch sizes are often chosen as a power of 2, i.e., 16,32,64,128,256 etc.
  3. Now, while choosing a proper size for mini-batch gradient descent, make sure that the minibatch fits in the CPU/GPU.
  4. 32 is generally a good choice

To know more, you can read this: A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.