XGBClassifier and RandomForestClassifier variable importance plots are very different

Question

I have one dataset, and decided to use XGBClassifier to get a variable importance plot from it.

I was a little surprised that some features that I assumed were insightful had no appreciable value. With this in mind, I used an sklearn RandomForestClassifier to create another variable importance plot.

To my surprise the two were very different! Many variables that appear on one don't appear on the other.

I used as similar parameters as I could, between the two models.

Here's what they look like to give a general idea:

Has anyone had this similar experience, and if so, what kinds of reasoning for this might there be? It's a little frightening to think that many Kagglers and others really depend on these plots and to see that they can vary so much...makes me wonder.

Thanks.

Answer 1

There are multiple reasons why this could happen. The first, and probably the most important one is that while both of the algorithms are tree based ensemble methods, they are very different from each other. Second of all, these feature importances are just some heuristic that do some calculations on where the features are used. I assume these heuristics are very different for both the algorithms

In this case however the differences might just be too big to only point to these two reasons. What could have happened is that your features are highly correlated. If two columns are highly correlated then the algorithms become kind of indifferent to which one to use, which means the two algorithms might both pick a different one each time they come across this choice. You could take a look if column 0 and column 7 are very correlated.

Another option is that some of your columns are categorical and that the implementations deal with this in a different way. Python doesn't work cleanly with categorical values in Scikit-learn. One-hot encoding is not a nice representation for undeep trees but if you believe this may be causing it you could preprocess them like that to see if the feature importances become more similar.