# Comparing Categorical and Continuous Features using Splits in GBM

In many GBM models you can get a rough feature importance of a feature by taking the number of splits done on that feature and comparing it to the splits on the other features. This works rather well until you get a mix of categorical and continuous features.

When you mix the continuous and categorical features the continuous features completely dominate as they have so many more places to split.

Is there a known method on how to compare the feature importance of continuous and categorical features from this kind of measure? If not, what suggestions do you have on doing this?

• I am not sure I agree with the premise. The number of splits that are possible doesn't affect how good they are, and this measure of feature importance depends on the number of times some one split on that feature was chosen as the best – Sean Owen Jul 14 '16 at 13:54
• @SeanOwen Let me be a little more specific. I am using xgboost get_fscore(). My understanding of the get_fscore() is as a result of reading this stackoverflow.com/questions/33652224/…. Therefore, in this case, it is implied that the more splits, the more important the feature right? – josh Jul 14 '16 at 14:24
• ... the more splits from that feature that were chosen as the best. The number of possible splits on that feature doesn't really matter for that. A continuous feature has basically infinite possible splits; a binary feature just 1, but, the binary feature's split could be better in most cases and be chosen as the split that goes in the tree more often. – Sean Owen Jul 14 '16 at 15:50
• Exactly. So it is not a great feature importance measure that could do with improving and so brings me back to "Is there a known method on how to compare the feature importance of continuous and categorical features?" – josh Jul 14 '16 at 16:29
• I think you're quite misunderstanding this measure of feature importance then.. not sure how else to explain it to you. – Sean Owen Jul 14 '16 at 21:57

For example, consider that you want to predict a continuous target $y$ from two features, $x_1, x_2$, where the ground truth is $y = x_1 + x_2$. Assume that $x_1$ can either be $0$ or $100$ and $x_2$ is a continuous variable ranging from $0$ to $1$. Training a model that fits exactly your training set will have only one split for $x_1$ and as many splits on $x_2$ than distinct training values of $x_2$. In this case, you can make $x_2$ arbitrarily more important that $x_1$ by increasing the number of distinct training points, but $x_2$ is mostly irrelevant to the accuracy of the prediction when compared to $x_1$. This problem makes it impossible to judge the importance of a feature based on the number of time it is used, in the general case.