In many GBM models you can get a rough feature importance of a feature by taking the number of splits done on that feature and comparing it to the splits on the other features. This works rather well until you get a mix of categorical and continuous features.

When you mix the continuous and categorical features the continuous features completely dominate as they have so many more places to split.

Is there a known method on how to compare the feature importance of continuous and categorical features from this kind of measure? If not, what suggestions do you have on doing this?

  • 2
    $\begingroup$ I am not sure I agree with the premise. The number of splits that are possible doesn't affect how good they are, and this measure of feature importance depends on the number of times some one split on that feature was chosen as the best $\endgroup$ – Sean Owen Jul 14 '16 at 13:54
  • $\begingroup$ @SeanOwen Let me be a little more specific. I am using xgboost get_fscore(). My understanding of the get_fscore() is as a result of reading this stackoverflow.com/questions/33652224/…. Therefore, in this case, it is implied that the more splits, the more important the feature right? $\endgroup$ – josh Jul 14 '16 at 14:24
  • $\begingroup$ ... the more splits from that feature that were chosen as the best. The number of possible splits on that feature doesn't really matter for that. A continuous feature has basically infinite possible splits; a binary feature just 1, but, the binary feature's split could be better in most cases and be chosen as the split that goes in the tree more often. $\endgroup$ – Sean Owen Jul 14 '16 at 15:50
  • $\begingroup$ Exactly. So it is not a great feature importance measure that could do with improving and so brings me back to "Is there a known method on how to compare the feature importance of continuous and categorical features?" $\endgroup$ – josh Jul 14 '16 at 16:29
  • $\begingroup$ I think you're quite misunderstanding this measure of feature importance then.. not sure how else to explain it to you. $\endgroup$ – Sean Owen Jul 14 '16 at 21:57

As pointed in the comments, there is a distinction to make between the importance of a feature for the model, i.e., how many times it is used, and the importance for the accuracy of the prediction. These two points might be unrelated.

For example, consider that you want to predict a continuous target $y$ from two features, $x_1, x_2$, where the ground truth is $y = x_1 + x_2$. Assume that $x_1$ can either be $0$ or $100$ and $x_2$ is a continuous variable ranging from $0$ to $1$. Training a model that fits exactly your training set will have only one split for $x_1$ and as many splits on $x_2$ than distinct training values of $x_2$. In this case, you can make $x_2$ arbitrarily more important that $x_1$ by increasing the number of distinct training points, but $x_2$ is mostly irrelevant to the accuracy of the prediction when compared to $x_1$. This problem makes it impossible to judge the importance of a feature based on the number of time it is used, in the general case.

I don't think there is a definite answer on how to rank features with respect to the accuracy either, as the information contained in one feature can be a duplicate from another one, or be useful only if you have another feature to complement it.

One simple method to approach this problem is the reverse of iterative feature selection, which you can read more on Feature Selection (Wikipedia). Given a model with 10 features, you can build 10 models with 9 features, removing a different one each time, and compute their loss on a validation set. If the 2nd model out of your 10 models performs the worst, then the 2nd feature is the most important. If the 9th model, without feature #9, is still as good as the full model, then feature #9 is not important, and you can use the difference in your loss metric to measure the importance to accuracy. However, it might be the case that feature #9 becomes useful if you add a new feature and that feature #2 can be built by a combination of other features, which you'd catch if you allowed your model to be more complex.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.