I'm using XGBoost and my mean absolute error on the validation set goes up when I change it from 0.05 to 0.03, I thought a smaller learning rate only makes it run slower and will if anything increase the accuracy of the model because all it does is make the step smaller, so if anything it's less prone to over shooting.
So maybe I don't know the learning rate as well as I thought.
Why does this happen?
PS. Is there a good way to figure out the learning rate and other parameters like n_estimators (in this case for XGBoost) and so on other than trial and error?