6
$\begingroup$

For some machine learning methods it is recommended to use feature normalization to use features that are on the same scale, especially for distance based methods like k-means or when using regularization. However, in my experience, boosting tree regression works less well when I use normalized features, for some strange reason. How is your experience using feature normalization with boosted trees does it in general improve our models?

$\endgroup$
5
$\begingroup$

Boosting trees is about building multiple decision trees. Decision tree doesn't require feature normalization, that's because the model only needs the absolute values for branching.

Wikipedia for decision tree:

Requires little data preparation. Other techniques often require data normalization....

However, it's always a good idea to normalize your features because:

  • It's easier to visualize and interpret your model
  • It's easier to compare another model (e.g. SVM) with the same data set
$\endgroup$
  • $\begingroup$ I thought the same. But doing feature normalization with boosted trees sometimes leads to worse models (in my experience). Therefore, it does not seem to be "always a good idea". I am asking more for experience than for general directions. $\endgroup$ – Sören Jan 13 '17 at 6:00
3
$\begingroup$

How is your experience using feature normalization with boosted trees does it in general improve our models?

My rather limited experience with scaling of features suggests that it has virtually no impact on xgboost results.

I suppose by normalisation you mean subtracting the mean and then dividing by standard deviation. If you calculated the statistics based on entire dataset (including holdout) you would get data leakage, which might indeed, at least theoretically, degrade the performance on holdout.

According to my understanding of xgboost, the correctly performed scaling should have no impact on the performance.

I suggest you double check your implementation or provide more details on how you do it, preferably with including a reproducible example.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.