4
$\begingroup$

I wonder about the difference between min_child_weight and gamma regularisation in XGBoost. From my understanding:

  • hessian regularisation blocks the individual trees ($f_t$) from growing (much like limiting max_depth)
  • loss regularisation blocks the individual trees ($f_t$) from being kept in the ensemble as it penalises number of trees

However, I have some troubles to find practical implications from that. I mean, is there a situation when hessian regularisation would perform better than gamma regularisation (or worse)? Or is it all dependent on the dataset and the other hyperparameter values? My feeling is that when my model loss is mainly caused by few awfully inaccurate prediction - then hessian regularisation might help more than gamma, and when my model is mistaken by approximetely the same amount for every observation then gamma would be better. But I'm struggle to figure out the proof for that feeling.

$\endgroup$
2
$\begingroup$

Loss regularization also prevents the trees from growing; it does not penalize the number of trees.

I think you're right that which is better (of course, you could use both) depends on the data and other hyperparameters. You can think though about whether a given node will split, depending on the type of regularization:
If you have a large node with some potential split that would create two other large nodes, but that split doesn't decrease the loss much, gamma will prevent it but min_child_weight won't. If you have a node with a potential split with one child being very small but that decreases the loss a lot, gamma will allow it but min_child_weight won't. So, personal opinion (based on that thought experiment, and not experience tuning these two together or against each other): set min_child_weight large enough that you're comfortable that a node with that many samples (in the squared-loss case, and appropriately transformed in other cases) is big enough to absorb noisiness of your population [by which I mean: one sample is surely quite likely to be random noise; an average of 100 might be better, but might not be enough, depending on your data]; after setting that, tune gamma. Better, tune both of them together, but still use the above intuition of your data to give a reasonable range for min_child_weight.

$\endgroup$
1
  • $\begingroup$ Does the range of tuned min_child_weight correlated with the number of feature or samples in the training dataset? What is the typical accepted range for gamma and min_child_weight parameters? $\endgroup$ Aug 3 '21 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.