36
$\begingroup$

I often read that in case of Deep Learning models the usual practice is to apply mini batches (generally a small one, 32/64) over several training epochs. I cannot really fathom the reason behind this.

Unless I'm mistaken, the batch size is the number of training instances let seen by the model during a training iteration; and epoch is a full turn when each of the training instances have been seen by the model. If so, I cannot see the advantage of iterate over an almost insignificant subset of the training instances several times in contrast with applying a "max batch" by expose all the available training instances in each turn to the model (assuming, of course, enough the memory). What is the advantage of this approach?

$\endgroup$
43
$\begingroup$

The key advantage of using minibatch as opposed to the full dataset goes back to the fundamental idea of stochastic gradient descent1.

In batch gradient descent, you compute the gradient over the entire dataset, averaging over potentially a vast amount of information. It takes lots of memory to do that. But the real handicap is the batch gradient trajectory land you in a bad spot (saddle point).

In pure SGD, on the other hand, you update your parameters by adding (minus sign) the gradient computed on a single instance of the dataset. Since it's based on one random data point, it's very noisy and may go off in a direction far from the batch gradient. However, the noisiness is exactly what you want in non-convex optimization, because it helps you escape from saddle points or local minima(Theorem 6 in [2]). The disadvantage is it's terribly inefficient and you need to loop over the entire dataset many times to find a good solution.

The minibatch methodology is a compromise that injects enough noise to each gradient update, while achieving a relative speedy convergence.

1 Bottou, L. (2010). Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010 (pp. 177-186). Physica-Verlag HD.

[2] Ge, R., Huang, F., Jin, C., & Yuan, Y. (2015, June). Escaping From Saddle Points-Online Stochastic Gradient for Tensor Decomposition. In COLT (pp. 797-842).

EDIT :

I just saw this comment on Yann LeCun's facebook, which gives a fresh perspective on this question (sorry don't know how to link to fb.)

Training with large minibatches is bad for your health. More importantly, it's bad for your test error. Friends dont let friends use minibatches larger than 32. Let's face it: the only people have switched to minibatch sizes larger than one since 2012 is because GPUs are inefficient for batch sizes smaller than 32. That's a terrible reason. It just means our hardware sucks.

He cited this paper which has just been posted on arXiv few days ago (Apr 2018), which is worth reading,

Dominic Masters, Carlo Luschi, Revisiting Small Batch Training for Deep Neural Networks, arXiv:1804.07612v1

From the abstract,

While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance ...

The best performance has been consistently obtained for mini-batch sizes between m=2 and m=32, which contrasts with recent work advocating the use of mini-batch sizes in the thousands.

$\endgroup$
  • 2
    $\begingroup$ Why should mini-batch gradient descent be more likely to avoid bad local minima than batch gradient descent? Do you have anything to back that claim up? $\endgroup$ – Martin Thoma Feb 9 '17 at 21:00
  • $\begingroup$ @MartinThoma See Theorem 6 in [2], a recent paper on JMLR. $\endgroup$ – horaceT Feb 11 '17 at 3:17
  • 2
    $\begingroup$ This paper is also on arXiv. Also, I don't see how this supports your claim. They never even mentioned mini-batch gradient descent. I do not understand that theorem (e.g. what is "g(X)"? Where did they introduce that notation? In statistics classes, g(X) = E(X)... but that doesn't make much sense here). What is $\phi(w, X)$? - The statement of this theorem seems to suggest that there are no bad local minima. But this would be true for SGD and batch gradient descent as well as mini-batch gradient descent, right? $\endgroup$ – Martin Thoma Feb 11 '17 at 7:25
  • 2
    $\begingroup$ @MartinThoma Given that there is one global minima for the dataset that we are given, the exact path to that global minima depends on different things for each GD method. For batch, the only stochastic aspect is the weights at initialization. The gradient path will be the same if you train the NN again with the same initial weights and dataset. For mini-batch and SGD, the path will have some stochastic aspects to it between each step from the stochastic sampling of data points for training at each step. This allows mini-batch and SGD to escape local optima if they are on the way. $\endgroup$ – Wesley Oct 29 '17 at 21:39
8
$\begingroup$

Memory is not really the reason for doing this, because you could just accumulate your gradients as you iterate through the dataset, and apply them at the end, but still in SGD you apply them at every step.

Reasons that SGD is used so widely are:

1) Efficiency. Typically, especially early in training, the parameter-gradients for different subsets of the data will tend to point in the same direction. So gradients evaluated on 1/100th of the data will point roughly in the same general direction as on the full dataset, but only require 1/100 the computation. Since convergence on a highly-nonlinear deep network typically requires thousands or millions of iterations no matter how good your gradients are, it makes sense to do many updates based on cheap estimates of the gradient rather than few updates based on good ones.

2) Optimization: Noisy updates may allow you to bounce out of bad local optima (though I don't have a source that shows that this matters in practice).

3) Generalization. It seems (see Zhang et al: Theory of Deep Learning III: Generalization Properties of SGD) that SGD actually helps generalization by finding "flat" minima on the training set, which are more likely to also be minima on the test set. Intuitively, we can think of SGD as a sort of Bagging - by computing our parameters based on many minibatches of the data, we reenforce rules that generalize across minibatches, and cancel rules that don't, thereby making us less prone to overfitting to the training set.

$\endgroup$
3
$\begingroup$

Unless I'm mistaken, the batch size is the number of training instances let seen by the model during a training iteration

Correct (although I would call it "weight update step")

and epoch is a full turn when each of the training instances have been seen by the model

Correct

If so, I cannot see the advantage of iterate over an almost insignificant subset of the training instances several times in contrast with applying a "max batch" by expose all the available training instances in each turn to the model (assuming, of course, enough the memory). What is the advantage of this approach?

Well, pretty much that. You usually don't have enough memory. Lets say we are talking about image classification. ImageNet is a wildly popular dataset. For quite a while, VGG-16D was one of the most popular mod.els. It needs calculcate 15 245 800 floats (in the feature maps) for one 224x224 image. This means about 61MB per image. This is just a rough lower bound on how much memory you need during training for each image. ImageNet contains several thousand (I think about 1.2 million?) images. While you might have that much main memory, you certainly do not have that much GPU memory. I've seen GPU speeding up things to about 21x. So you definitely want to use the GPU.

Also: The time for one mini-batch is much lower. So the question is: Would you rather do n update steps with mini-batch per hour on a GPU or m update steps with batch without GPU, where n >> m.

$\endgroup$
  • $\begingroup$ It's really not a matter of limited memory. It's always possible to compute the gradient over your dataset in a number of batches with fixed model parameters (functionally equivalent to computing the gradient in a single giant batch). Practically it's more about the generalization properties caused by stochasticity / noisiness of SGD / MBGD and the fact that fewer epochs through your dataset are required in order to reach convergence. Updating model params within a single epoch leads to better intermediate params which makes further gradient calculations within the epoch more informative. $\endgroup$ – Madison May May 8 '18 at 22:00
1
$\begingroup$

Aside from the other answers I think it's worth pointing out that there are two quantities which are distinct but often coupled:

  1. The number of inputs used to compute the gradient of the parameters in each step.

As others have pointed out, the gradient with respect to a minibatch is an approximation of the true gradient. The larger the minibatch, the better the approximation.

  1. The number of inputs collected into an array and computed "at the same time"

The trade off here is purely about performance (memory/cycles).

These quantities are typically the same, i.e. the minibatch size, but in principle they can be decoupled.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.