I found this term "training warmup steps" in some of the papers, what exactly does this term mean? Has it got anything to do with "learning rate"? If so, how does it affect?
This usually means that you use a very low learning rate for a set number of training steps (warmup steps). After your warmup steps you use your "regular" learning rate or learning rate scheduler. You can also gradually increase your learning rate over the number of warmup steps.
As far as I know, this has the benefit of slowly starting to tune things like attention mechanisms in your network.
Warm up steps is just a parameter in most of the learning algorithms which is used to lower the learning rate in order to reduce the impact of deviating the model from learning on sudden new data set exposure.
For eg:- If you are giving warm up steps as 500 for a iteration of 10,000 epochs, For the first 500 iterations the model will learn the corpus with minimal learning rate than the rate which you've specified in the model. From 501 th iteration model will use the learning rate as itself which given.