1
$\begingroup$

Many expert said "Batch has more local optimal possibility than SGD". But, I don't know the reason..

enter image description here

How SGD could avoid local optimal better than Batch? (Some people tell me about over shooting as the reason. But, I don't also know there is more over shooting in SGD)

If you explain this reason by using Geometric, Mathmatics or other intuition concept, It's gonna be great!

$\endgroup$
1
$\begingroup$

enter image description here

For gradient descent, batch updates calculate the weights based on all examples. This process continues until convergence which is typically measured by a derivative of zero. The resulting weights might end up in a local minimum or saddle point because both have a derivative of zero.

Stochastic gradient descent takes a random sampling of examples, introducing randomness into the updates. This randomness increases the possibilities of escaping a local minimum or saddle point. The updates might randomly overestimate the area with a derivative of zero and find a new section of the error surface that leads to a better optimum.

$\endgroup$
0
$\begingroup$

For the purpose of this answer, let’s assume your training data is more or less perfectly representative of the distribution of the real world data it was sampled from.

Let’s think about mini batch gradient descent. When you have a small batch size, each training step uses a small batch of training examples which is probably not able to accurately represent the wider data set. This means the calculated error gradient is not quite accurate, and effectively the parameters of the model are updated as though the small set of training examples is totally representative of the underlying distribution. If the mini batch is especially far from the underlying distribution, the mini batches that come after it will on average bring the model back towards the correct distribution.

We use a small learning rate to ensure that only small updates are made at each training step, which essentially reduces the risk of the model parameters going wildly off course in response to a weird mini batch and therefore keeps the model’s descent nearer to an imaginary “perfect” descent down the error gradient (because on average a mini batch will represent the exact distribution of your training data).

If we use bigger batches, they are more likely to be representative of the underlying distribution, so we can safely use a higher learning rate. As our model starts to converge, it should still keep moving towards a true error minimum and not be thrown off by weird batches. We might use learning rate scheduling to reduce the learning rate as training goes on.

Stochastic gradient descent is equivalent to mini batch gradient descent with batch size 1, so we’re in danger of being thrown off our ideal descent by weird observations if our learning rate is too high. This might throw us off the path to a local minimum and onto a path towards the global minimum (or at least a better local minimum), but it might also do the opposite. Therefore, this is not a good justification of using a very small batch size.

Of course, sometimes we’re forced to use small batch sizes because our data is too big to fit many observations in memory at once. Otherwise, a bigger batch size is better.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.