0
$\begingroup$

As far as I know, mini-batch can be used to reduce the variance of the gradient, but I am also considering if we can achieve the same result if we use the decreasing step size and only single sample in each iteration? Can we compare the convergence rate of them?

$\endgroup$
0
$\begingroup$

Generally answer is "it's not known". Similarity of effects of increasing minibatches size and decreasing learning rate is mostly empirical, there is no known asymptotic formula for it. Also effect of small LR and big minibatch is not the same. For example batch normalization layer would act completely different on those two approaches. Probabilistic distribution of gradients produced by minibatches and single sample (or mb of significantly different size) would be also quite different

| improve this answer | |
$\endgroup$
  • $\begingroup$ As @mirror2image says - there's no free lunch for you. It depends highly on your data, domain, problem - run your experiments and check it. Generally speaking, rarely there's some significant a priori knowledge. Running many experiments will also aid your intuition for future research, as a space of hyper-parameters you can research is always limited. $\endgroup$ – Piotr Rarus - Reinstate Monica Nov 25 '19 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.