0
$\begingroup$

I thought the consensus was that XGBoost was largely scale-invariant and scaling of features isn't really necessary but something's going wrong and I don't understand what.

I have a range of features with different scales and I'm trying to do a regression to target a variable that's in the range of 1E-7 or so (i.e. the target will be somewhere between 1E-7 and 9E-7).

When I run XGBoost on this, I get warnings about "0 depth" trees and every prediction is the same value, regardless of the value of the input feature.

If I scale my target variable by 1E9 (so it's on the order of 100 rather than 1E-7) then the regression works perfectly fine and I get a decent model.

Can anyone shed any light on this? Is there a particular hyperparameter that's sensitive to the scale of the target variable?

$\endgroup$

1 Answer 1

2
$\begingroup$

It may not be related to hyperparams per say. I think it has more to do with the nature of how xgboost is trained. XGBoost for regression tries to reduce the variance at every node. May when you have variables with such less values it just round of them to 0 and it has nothing to learn. It may have to do more with precision which XGBoost work.

$\endgroup$
2
  • $\begingroup$ Thanks! I was thinking it was something like that but I assumed somewhere there's a hyperparameter that describes what "insignificant" is. At the minute there's so many tuning options that it's hard to spot what that might be. $\endgroup$
    – Rob
    Apr 6, 2022 at 13:31
  • 1
    $\begingroup$ In particular, the mean squared error is being minimized; the squared errors for those targets will be <~10^(-14), which might overflow numerical precision. $\endgroup$
    – Ben Reiniger
    Apr 6, 2022 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.