43
$\begingroup$

I found the term "training warmup steps" in some of the papers. What exactly does this term mean? Has it got anything to do with "learning rate"? If so, how does it affect it?

$\endgroup$

5 Answers 5

31
$\begingroup$

This usually means that you use a very low learning rate for a set number of training steps (warmup steps). After your warmup steps you use your "regular" learning rate or learning rate scheduler. You can also gradually increase your learning rate over the number of warmup steps.

As far as I know, this has the benefit of slowly starting to tune things like attention mechanisms in your network.

$\endgroup$
2
  • 6
    $\begingroup$ But isn't it the normal trend to use learning rate decay, which reduces the learning rate over sent of epochs or training steps? $\endgroup$ Commented Jul 19, 2019 at 14:43
  • 3
    $\begingroup$ Yes. The idea is to use some warmup steps to increase the learning rate up to a certain point and then use your normal learning rate decay afterwards. The google transformer notebook has aa good example in the optimizer setting: tensorflow.org/beta/tutorials/text/transformer#optimizer $\endgroup$ Commented Jul 20, 2019 at 16:25
38
$\begingroup$

As the other answers already state: Warmup steps are just a few updates with low learning rate before / at the beginning of training. After this warmup, you use the regular learning rate (schedule) to train your model to convergence.

The idea that this helps your network to slowly adapt to the data intuitively makes sense. However, theoretically, the main reason for warmup steps is to allow adaptive optimisers (e.g. Adam, RMSProp, ...) to compute correct statistics of the gradients. Therefore, a warmup period makes little sense when training with plain SGD.

E.g. RMSProp computes a moving average of the squared gradients to get an estimate of the variance in the gradients for each parameter. For the first update, the estimated variance is just the square root of the sum of the squared gradients for the first batch. Since, in general, this will not be a good estimate, your first update could push your network in a wrong direction. To avoid this problem, you give the optimiser a few steps to estimate the variance while making as little changes as possible (low learning rate) and only when the estimate is reasonable, you use the actual (high) learning rate.


Add-on

As Michael pointed out, my answer might suggest that running one epoch with zero learning rate should do the job. However, there are a few reasons why this might not work as well:

  1. One epoch with lr=0 is just a lost epoch in the end. If you're iterating through the data you can as well have a very small learning rate to do at least something.
  2. After this first "useless" epoch, you would have accurate statistics. However, if you directly jump in with a high learning rate, your network (and therefore also the gradients) might change a lot. Therefore, you want to slowly increase the learning rate so that the statistics get a chance to move along with the updates.

Disclaimer: these arguments are intuitions/speculations rather than plain facts. Feel free to let me know if this does work in practice!

$\endgroup$
5
  • $\begingroup$ it might be useful to point out that SGD with momentum is also to be considered adaptive in this context... $\endgroup$ Commented Oct 7, 2021 at 12:03
  • 1
    $\begingroup$ Then why not just make the first epoch use lr=0, then continue as normal? I.e. why multiple epochs are needed, and why positive learning rate is needed? $\endgroup$ Commented Oct 27, 2021 at 15:24
  • 1
    $\begingroup$ @MichaelLitvin I extended the answer to address your comment. Hope it helps clear things out. $\endgroup$ Commented Oct 28, 2021 at 6:27
  • $\begingroup$ I would like to add to this question of lr=0. Optimizers like Adam also make use of momentum. If you keep lr=0, there would be no momentum in the gradient descent. Like in the world of Physics, momentum is responsible for making it harder for the motion of descent to change the direction of descent. During warmup, the gradient descent gathers some momentum so that at the end of the warmup, it would be "propelled" in a more correct direction. If lr=0 for initial epoch/s there is no real advantage, it would be just like running GD without any warmup because momentum would be 0. $\endgroup$
    – FutureJJ
    Commented Nov 8, 2023 at 3:05
  • $\begingroup$ In my opinion, this answer is not 100% correct. In gradient descent, the initial samples matter more than later samples (There is a paper by Joshua Bengio et al. on it). Not only your data matters but the order of data in opimization. Low learning rate in the beginning will reduce variance in the results which may be desirable. $\endgroup$
    – Servus
    Commented Jun 13 at 9:38
4
$\begingroup$

Warm up steps is just a parameter in most of the learning algorithms which is used to lower the learning rate in order to reduce the impact of deviating the model from learning on sudden new data set exposure.

For eg:- If you are giving warm up steps as 500 for a iteration of 10,000 epochs, For the first 500 iterations the model will learn the corpus with minimal learning rate than the rate which you've specified in the model. From 501 th iteration model will use the learning rate as itself which given.

$\endgroup$
4
$\begingroup$

If your data set is highly differentiated, you can suffer from a sort of "early over-fitting". If your shuffled data happens to include a cluster of related, strongly-featured observations, your model's initial training can skew badly toward those features -- or worse, toward incidental features that aren't truly related to the topic at all.

Warm-up is a way to reduce the primacy effect of the early training examples. Without it, you may need to run a few extra epochs to get the convergence desired, as the model un-trains those early superstitions.

Many models afford this as a command-line option. The learning rate is increased linearly over the warm-up period. If the target learning rate is p and the warm-up period is n, then the first batch iteration uses 1p/n for its learning rate; the second uses 2p/n, and so on: iteration i uses i*p/n, until we hit the nominal rate at iteration n.

This means that the first iteration gets only 1/n of the primacy effect. This does a reasonable job of balancing that influence.

Note that the ramp-up is commonly on the order of one epoch -- but is occasionally longer for particularly skewed data, or shorter for more homogeneous distributions. You may want to adjust, depending on how functionally extreme your batches can become when the shuffling algorithm is applied to the training set.

source

$\endgroup$
4
$\begingroup$

It can assume also other meaning but the learning rate schedule process. For example in YOLOv3, during warm up epochs the ground truth bounding boxes are forced to be the same size of the anchors.

At the end of the day, the warm up procedure aim to soften the impact of the first epochs of learning that can mislead the entire training process. This is not mathematically proven (as far as I know), it just a solid intuition that actually happens to result in better performances.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.