It is believed by many that tree-based methods are invariant under monotone transformations of the predictors. But recently I've read a paper (https://arxiv.org/pdf/1611.04561.pdf, referred to as the arxiv paper later) that says whether it's invariant depends on how the split threshold is chosen (there are three methods), and according to this paper, xgboost would be invariant under transformations because it uses the sweep left method. This is mentioned in the last paragraph on pp.2 and first paragraph on pp.3.

But when I read the original xgboost paper by Chen, the split algorithm looks much more sophisticated than any method mentioned in the arxiv paper, and it looks like it should be sensitive to transformations. I've tried xgboost on regression for a few data sets, and if I have column subsampling turned on, I do see different results with predictor transformations.

Can anyone give me some confirmation on this topic? I'm confused mostly by the arxiv paper.

  • 1
    $\begingroup$ Do you confirm that making no change also has no change in your outputs? You may be seeing randomness changing things. $\endgroup$ – kbrose Jun 4 '18 at 22:45
  • 1
    $\begingroup$ Yes. Seed is set, and results don't change if I run multiple times with the same set of parameters. $\endgroup$ – hooyeh Jun 5 '18 at 0:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.