1
$\begingroup$

We know that when a batch size is too large, the model might not be able to converge. But what is the drawback of having a batch size be too small (say batch_size = 1) other than taking a long time to train? A batch size of 32 is commonly used and referred to as "small," but why don't we use a smaller batch size of 1 to guarantee convergence?

$\endgroup$
2
  • $\begingroup$ It will be useful if you could give some more information regarding your NN. $\endgroup$ May 30 '19 at 11:44
  • $\begingroup$ For getting a better grasp of practical consideration in gradient descent, I generally advise to read the article "efficient" back prop by lecun. $\endgroup$
    – lcrmorin
    Feb 25 '20 at 17:39
7
$\begingroup$

A batch size of 32 is commonly used and referred to as "small," but why don't we use a smaller batch size of 1 to guarantee convergence?

The issue is that a small batch size both helps and hurts convergence. Updating the weights based on a small batch will be more noisy. The noise can be good, helping by jerking out of local optima. However, the same noise and jerkiness will prevent the descent from fully converging to an optima at all.

It is a trade off between these factors, the best batch size will depend on the shape of the error manifold. Larger batch sizes are better on convex errors and smaller batch size are good on errors with lots of deeper local optima.

convergence

$\endgroup$
1
  • 1
    $\begingroup$ explainable picture!! $\endgroup$
    – Shark Deng
    Jun 27 '20 at 2:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.