Lowering learning rate makes my accuracy on the validation set go down

Question

I'm using XGBoost and my mean absolute error on the validation set goes up when I change it from 0.05 to 0.03, I thought a smaller learning rate only makes it run slower and will if anything increase the accuracy of the model because all it does is make the step smaller, so if anything it's less prone to over shooting.

So maybe I don't know the learning rate as well as I thought.

Why does this happen?

PS. Is there a good way to figure out the learning rate and other parameters like n_estimators (in this case for XGBoost) and so on other than trial and error?

Answer 1

Optimizer functions in ML algorithms update themselves with learning rate to converge local minimums. When the step size (here learning rate = eta) gets smaller the function may not converge since there are not enough steps with this small learning rate (step size).

You should increase your learning rate or number of steps while keeping the learning rate constant to deal with the problem.

You may want to look at gradient descent: https://en.wikipedia.org/wiki/Gradient_descent

Also following image explains very well how to tune learning rate. (source: https://www.jeremyjordan.me/nn-learning-rate/)

Answer 2

Make sure to increase the value of n_estimators when you decrease the learning_rate, otherwise the model will stop too early to find the optimal values. This should be done after tuning the other parameters.

Here are a couple of good tutorials regarding the tuning of the hyperparameters:

https://blog.cambridgespark.com/hyperparameter-tuning-in-xgboost-4ff9100a3b2f

https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/