For some machine learning methods it is recommended to use feature normalization to use features that are on the same scale, especially for distance based methods like k-means or when using regularization. However, in my experience, boosting tree regression works less well when I use normalized features, for some strange reason. How is your experience using feature normalization with boosted trees does it in general improve our models?
Boosting trees is about building multiple decision trees. Decision tree doesn't require feature normalization, that's because the model only needs the absolute values for branching.
Requires little data preparation. Other techniques often require data normalization....
However, it's always a good idea to normalize your features because:
- It's easier to visualize and interpret your model
- It's easier to compare another model (e.g. SVM) with the same data set
How is your experience using feature normalization with boosted trees does it in general improve our models?
My rather limited experience with scaling of features suggests that it has virtually no impact on xgboost results.
I suppose by normalisation you mean subtracting the mean and then dividing by standard deviation. If you calculated the statistics based on entire dataset (including holdout) you would get data leakage, which might indeed, at least theoretically, degrade the performance on holdout.
According to my understanding of xgboost, the correctly performed scaling should have no impact on the performance.
I suggest you double check your implementation or provide more details on how you do it, preferably with including a reproducible example.
Sometimes feature scaling/normalization is usefully even for tree-based model. For example, assume , where denotes the mean value of feature x in a time period T, if you don't do feature scaling/normalization, your model may perform poorly if the distribution of x changes in the future , and time-series normalization will help in this case.
Besides, if you are using Trees with linear models at leaves for lightgbm, it's important to scale the data.