8
$\begingroup$

I found this term "training warmup steps" in some of the papers, what exactly does this term mean? Has it got anything to do with "learning rate"? If so, how does it affect?

$\endgroup$
6
$\begingroup$

This usually means that you use a very low learning rate for a set number of training steps (warmup steps). After your warmup steps you use your "regular" learning rate or learning rate scheduler. You can also gradually increase your learning rate over the number of warmup steps.

As far as I know, this has the benefit of slowly starting to tune things like attention mechanisms in your network.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ But isn't it the normal trend to use learning rate decay, which reduces the learning rate over sent of epochs or training steps? $\endgroup$ – Ashwin Geet D'Sa Jul 19 '19 at 14:43
  • 1
    $\begingroup$ Yes. The idea is to use some warmup steps to increase the learning rate up to a certain point and then use your normal learning rate decay afterwards. The google transformer notebook has aa good example in the optimizer setting: tensorflow.org/beta/tutorials/text/transformer#optimizer $\endgroup$ – Ron Schwessinger Jul 20 '19 at 16:25
6
$\begingroup$

As the other answers already state: Warmup steps are just a few updates with low learning rate before / at the beginning of training. After this warmup, you use the regular learning rate (schedule) to train your model to convergence.

The idea that this helps your network to slowly adapt to the data intuitively makes sense. However, theoretically, the main reason for warmup steps is to allow adaptive optimisers (e.g. Adam, RMSProp, ...) to compute correct statistics of the gradients. Therefore, a warmup period makes little sense when training with plain SGD.

E.g. RMSProp computes a moving average of the squared gradients to get an estimate of the variance in the gradients for each parameter. For the first update, the estimated variance is just the square root of the sum of the squared gradients for the first batch. Since, in general, this will not be a good estimate, your first update could push your network in a wrong direction. To avoid this problem, you give the optimiser a few steps to estimate the variance while making as little changes as possible (low learning rate) and only when the estimate is reasonable, you use the actual (high) learning rate.

| improve this answer | |
$\endgroup$
2
$\begingroup$

Warm up steps is just a parameter in most of the learning algorithms which is used to lower the learning rate in order to reduce the impact of deviating the model from learning on sudden new data set exposure.

For eg:- If you are giving warm up steps as 500 for a iteration of 10,000 epochs, For the first 500 iterations the model will learn the corpus with minimal learning rate than the rate which you've specified in the model. From 501 th iteration model will use the learning rate as itself which given.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.