0
$\begingroup$

I have a problem, for simplicity let's say it is a binary classification problem.

I am trying to solve this problem using XGBoost.

A standard output plot for any ML algorithm, is the feature importance, and I want to look at the top n features.

But how to decide on n ?

A method would be to add a random variable (a vector with numbers from uniform distribution) in the XGBoost model, and then see in which place this random variable lands, and only look at the variables above the random variable.

If I do that, and then I re-run the model, including only the variables that score above the random one, then the accuracy score in the test set, decreases. Why ?

$\endgroup$
1
$\begingroup$

I'll go through your questions one by one:

... how to decide on n ?

This is a completely arbitrary choice that depends on your subjective preferences, the nature of your data, your problem at hand, the amount of computational power you can afford, you name it... There is no rule-of-thumb on this. What is the purpose of your analysis?

--

If I do that, and then I re-run the model, including only the variables that score above the random one, then the accuracy score in the test set, decreases. Why ?

The variables that have an explanatory power below a given threshold are comparatively less relevant but not useless, they can capture some of the variance of your dependent variable. Removing them means renouncing to a little bit of predictive power. That's why accuracy decreases.

Hope this helps, otherwise let me know.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.