Why mini batch size is better than one single "batch" with all training data?

Question

I often read that in case of Deep Learning models the usual practice is to apply mini batches (generally a small one, 32/64) over several training epochs. I cannot really fathom the reason behind this.

Unless I'm mistaken, the batch size is the number of training instances let seen by the model during a training iteration; and epoch is a full turn when each of the training instances have been seen by the model. If so, I cannot see the advantage of iterate over an almost insignificant subset of the training instances several times in contrast with applying a "max batch" by expose all the available training instances in each turn to the model (assuming, of course, enough the memory). What is the advantage of this approach?

Answer 1

The key advantage of using minibatch as opposed to the full dataset goes back to the fundamental idea of stochastic gradient descent1.

In batch gradient descent, you compute the gradient over the entire dataset, averaging over potentially a vast amount of information. It takes lots of memory to do that. But the real handicap is the batch gradient trajectory land you in a bad spot (saddle point).

In pure SGD, on the other hand, you update your parameters by adding (minus sign) the gradient computed on a single instance of the dataset. Since it's based on one random data point, it's very noisy and may go off in a direction far from the batch gradient. However, the noisiness is exactly what you want in non-convex optimization, because it helps you escape from saddle points or local minima(Theorem 6 in [2]). The disadvantage is it's terribly inefficient and you need to loop over the entire dataset many times to find a good solution.

The minibatch methodology is a compromise that injects enough noise to each gradient update, while achieving a relative speedy convergence.

1 Bottou, L. (2010). Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010 (pp. 177-186). Physica-Verlag HD.

[2] Ge, R., Huang, F., Jin, C., & Yuan, Y. (2015, June). Escaping From Saddle Points-Online Stochastic Gradient for Tensor Decomposition. In COLT (pp. 797-842).

EDIT :

I just saw this comment on Yann LeCun's facebook, which gives a fresh perspective on this question (sorry don't know how to link to fb.)

Training with large minibatches is bad for your health. More importantly, it's bad for your test error. Friends dont let friends use minibatches larger than 32. Let's face it: the only people have switched to minibatch sizes larger than one since 2012 is because GPUs are inefficient for batch sizes smaller than 32. That's a terrible reason. It just means our hardware sucks.

He cited this paper which has just been posted on arXiv few days ago (Apr 2018), which is worth reading,

Dominic Masters, Carlo Luschi, Revisiting Small Batch Training for Deep Neural Networks, arXiv:1804.07612v1

From the abstract,

While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance ...

The best performance has been consistently obtained for mini-batch sizes between m=2 and m=32, which contrasts with recent work advocating the use of mini-batch sizes in the thousands.

Answer 2

Memory is not really the reason for doing this, because you could just accumulate your gradients as you iterate through the dataset, and apply them at the end, but still in SGD you apply them at every step.

Reasons that SGD is used so widely are:

1) Efficiency. Typically, especially early in training, the parameter-gradients for different subsets of the data will tend to point in the same direction. So gradients evaluated on 1/100th of the data will point roughly in the same general direction as on the full dataset, but only require 1/100 the computation. Since convergence on a highly-nonlinear deep network typically requires thousands or millions of iterations no matter how good your gradients are, it makes sense to do many updates based on cheap estimates of the gradient rather than few updates based on good ones.

2) Optimization: Noisy updates may allow you to bounce out of bad local optima (though I don't have a source that shows that this matters in practice).

3) Generalization. It seems (see Zhang et al: Theory of Deep Learning III: Generalization Properties of SGD) that SGD actually helps generalization by finding "flat" minima on the training set, which are more likely to also be minima on the test set. Intuitively, we can think of SGD as a sort of Bagging - by computing our parameters based on many minibatches of the data, we reenforce rules that generalize across minibatches, and cancel rules that don't, thereby making us less prone to overfitting to the training set.

Answer 3

Unless I'm mistaken, the batch size is the number of training instances let seen by the model during a training iteration

Correct (although I would call it "weight update step")

and epoch is a full turn when each of the training instances have been seen by the model

Correct

If so, I cannot see the advantage of iterate over an almost insignificant subset of the training instances several times in contrast with applying a "max batch" by expose all the available training instances in each turn to the model (assuming, of course, enough the memory). What is the advantage of this approach?

Well, pretty much that. You usually don't have enough memory. Lets say we are talking about image classification. ImageNet is a wildly popular dataset. For quite a while, VGG-16D was one of the most popular mod.els. It needs calculcate 15 245 800 floats (in the feature maps) for one 224x224 image. This means about 61MB per image. This is just a rough lower bound on how much memory you need during training for each image. ImageNet contains several thousand (I think about 1.2 million?) images. While you might have that much main memory, you certainly do not have that much GPU memory. I've seen GPU speeding up things to about 21x. So you definitely want to use the GPU.

Also: The time for one mini-batch is much lower. So the question is: Would you rather do n update steps with mini-batch per hour on a GPU or m update steps with batch without GPU, where n >> m.

Answer 4

Aside from the other answers I think it's worth pointing out that there are two quantities which are distinct but often coupled:

1. The number of inputs used to compute the gradient of the parameters in each step.

As others have pointed out, the gradient with respect to a minibatch is an approximation of the true gradient. The larger the minibatch, the better the approximation.

2. The number of inputs collected into an array and computed "at the same time"

The trade off here is purely about performance (memory/cycles).

These quantities are typically the same, i.e. the minibatch size, but in principle they can be decoupled.

Answer 5

Mini-Batch Gradient Descent(MBGD):

• Pros:
  • It's good at a large dataset while BGD is not.
  • It's good at online learning while BGD is not. *Online learning is the way which a model incrementally learns from a stream of dataset in real-time.
  • It doesn't need the repreparation of a whole dataset if you want to update a model while BGD needs.
  • It can more easily escape local minima or saddle points than BGD.
• Cons:
  • The computation is less stable than BGD.
  • It's less strong in noise(noisy data) than BGD.
  • It gets a less accurate value than BGD.