I wonder about the difference between
gamma regularisation in XGBoost. From my understanding:
- hessian regularisation blocks the individual trees ($f_t$) from growing (much like limiting
- loss regularisation blocks the individual trees ($f_t$) from being kept in the ensemble as it penalises number of trees
However, I have some troubles to find practical implications from that. I mean, is there a situation when hessian regularisation would perform better than gamma regularisation (or worse)? Or is it all dependent on the dataset and the other hyperparameter values? My feeling is that when my model loss is mainly caused by few awfully inaccurate prediction - then hessian regularisation might help more than gamma, and when my model is mistaken by approximetely the same amount for every observation then gamma would be better. But I'm struggle to figure out the proof for that feeling.