How to interpret a random variable in the variable importance?

Question

I have a problem, for simplicity let's say it is a binary classification problem.

I am trying to solve this problem using XGBoost.

A standard output plot for any ML algorithm, is the feature importance, and I want to look at the top n features.

But how to decide on n ?

A method would be to add a random variable (a vector with numbers from uniform distribution) in the XGBoost model, and then see in which place this random variable lands, and only look at the variables above the random variable.

If I do that, and then I re-run the model, including only the variables that score above the random one, then the accuracy score in the test set, decreases. Why ?

Answer 1

I'll go through your questions one by one:

... how to decide on n ?

This is a completely arbitrary choice that depends on your subjective preferences, the nature of your data, your problem at hand, the amount of computational power you can afford, you name it... There is no rule-of-thumb on this. What is the purpose of your analysis?

--

If I do that, and then I re-run the model, including only the variables that score above the random one, then the accuracy score in the test set, decreases. Why ?

The variables that have an explanatory power below a given threshold are comparatively less relevant but not useless, they can capture some of the variance of your dependent variable. Removing them means renouncing to a little bit of predictive power. That's why accuracy decreases.

Hope this helps, otherwise let me know.