We know that when a batch size is too large, the model might not be able to converge. But what is the drawback of having a batch size be too small (say batch_size = 1) other than taking a long time to train? A batch size of 32 is commonly used and referred to as "small," but why don't we use a smaller batch size of 1 to guarantee convergence?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ It will be useful if you could give some more information regarding your NN. $\endgroup$– Shubham PanchalMay 30, 2019 at 11:44
-
$\begingroup$ For getting a better grasp of practical consideration in gradient descent, I generally advise to read the article "efficient" back prop by lecun. $\endgroup$– lcrmorinFeb 25, 2020 at 17:39
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
A batch size of 32 is commonly used and referred to as "small," but why don't we use a smaller batch size of 1 to guarantee convergence?
The issue is that a small batch size both helps and hurts convergence. Updating the weights based on a small batch will be more noisy. The noise can be good, helping by jerking out of local optima. However, the same noise and jerkiness will prevent the descent from fully converging to an optima at all.
It is a trade off between these factors, the best batch size will depend on the shape of the error manifold. Larger batch sizes are better on convex errors and smaller batch size are good on errors with lots of deeper local optima.
answered May 30, 2019 at 5:59
-
3