For some machine learning methods it is recommended to use feature normalization to use features that are on the same scale, especially for distance based methods like k-means or when using regularization. However, in my experience, boosting tree regression works less well when I use normalized features, for some strange reason. How is your experience using feature normalization with boosted trees does it in general improve our models?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Is it common to boost random forrest? (And if so, do you happen to have any literature on it?). $\endgroup$– S van BalenCommented Jan 10, 2017 at 11:32
-
1$\begingroup$ XGBoost: A Scalable Tree Boosting System $\endgroup$– SoerendipCommented Jan 13, 2017 at 5:57
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
Boosting trees is about building multiple decision trees. Decision tree doesn't require feature normalization, that's because the model only needs the absolute values for branching.
Requires little data preparation. Other techniques often require data normalization....
However, it's always a good idea to normalize your features because:
- It's easier to visualize and interpret your model
- It's easier to compare another model (e.g. SVM) with the same data set
answered Jan 10, 2017 at 10:02
-
2$\begingroup$ I thought the same. But doing feature normalization with boosted trees sometimes leads to worse models (in my experience). Therefore, it does not seem to be "always a good idea". I am asking more for experience than for general directions. $\endgroup$ Commented Jan 13, 2017 at 6:00
$\begingroup$
$\endgroup$
How is your experience using feature normalization with boosted trees does it in general improve our models?
My rather limited experience with scaling of features suggests that it has virtually no impact on xgboost results.
I suppose by normalisation you mean subtracting the mean and then dividing by standard deviation. If you calculated the statistics based on entire dataset (including holdout) you would get data leakage, which might indeed, at least theoretically, degrade the performance on holdout.
According to my understanding of xgboost, the correctly performed scaling should have no impact on the performance.
I suggest you double check your implementation or provide more details on how you do it, preferably with including a reproducible example.
$\begingroup$
$\endgroup$
Sometimes feature scaling/normalization is usefully even for tree-based model. For example, assume , where
denotes the mean value of feature x in a time period T, if you don't do feature scaling/normalization, your model may perform poorly if the distribution of x changes in the future
, and time-series normalization will help in this case.
Besides, if you are using Trees with linear models at leaves for lightgbm, it's important to scale the data.
