I have noticed that when you make a small decision tree model, and then extend the model by creating an ensemble of trees around the same tree settings, the variable importance is diluted in the sense that the least important and most important variables become a lot more closer together. In some cases, there may be almost do distinction in importance. Are there methods available to either mitigate this effect or to measure it, with a view to defining any tradeoff between understanding variable importance and overall accuracy?
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ how are you measuring "importance"? $\endgroup$– mandataJun 16, 2016 at 4:29
-
1$\begingroup$ Single decision trees tend to overfit the data - that might be the reason why you see a bigger gap between the most and least important variables - while in reality the identified variables might just contain noise that the single decision tree has fit to. The "diluted" (averaged) variable importances obtained from decision tree ensembles (like random forests) might actually be more realistic. Just guessing though.. $\endgroup$– stmaxJun 16, 2016 at 8:10
-
$\begingroup$ It is unclear how you measure "importance". Random forests provide a measure of variable importance, decision tree does not provide a sorted list of variable importance afaik. $\endgroup$– gc5Aug 15, 2016 at 19:42
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I've been using tree-based enesembling methods such as random forests and gradient boosting for several years now, and I have to say that I've never seen that behavior.
Some packages measure variable importance solely based on trees' final splits rather than candidate/surrogate splits, so if you have two important inputs that are correlated, but one is consistently a tiny bit better than the other, the less important input might never get selected as a final split and thus look as important as some of the less valuable inputs. However, this phenomenon is independent of the number of trees/ensembling, so I don't think it fully explains the behavior you're seeing.
answered Jun 16, 2016 at 18:26