3
$\begingroup$

We know that when a batch size is too large, the model might not be able to converge. But what is the drawback of having a batch size be too small (say batch_size = 1) other than taking a long time to train? A batch size of 32 is commonly used and referred to as "small," but why don't we use a smaller batch size of 1 to guarantee convergence?

$\endgroup$
2
  • $\begingroup$ It will be useful if you could give some more information regarding your NN. $\endgroup$ May 30, 2019 at 11:44
  • $\begingroup$ For getting a better grasp of practical consideration in gradient descent, I generally advise to read the article "efficient" back prop by lecun. $\endgroup$ Feb 25, 2020 at 17:39

1 Answer 1

12
$\begingroup$

A batch size of 32 is commonly used and referred to as "small," but why don't we use a smaller batch size of 1 to guarantee convergence?

The issue is that a small batch size both helps and hurts convergence. Updating the weights based on a small batch will be more noisy. The noise can be good, helping by jerking out of local optima. However, the same noise and jerkiness will prevent the descent from fully converging to an optima at all.

It is a trade off between these factors, the best batch size will depend on the shape of the error manifold. Larger batch sizes are better on convex errors and smaller batch size are good on errors with lots of deeper local optima.

convergence

$\endgroup$
1
  • 6
    $\begingroup$ explainable picture!! $\endgroup$
    – Shark Deng
    Jun 27, 2020 at 2:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.