9
$\begingroup$

I was working on a small classification problem (breast cancer data set from sklearn), and trying to decide which features were most important to predict the labels. I understand that there are several ways to define "important feature" here (permutation importance, importance in trees ...), but I did the following: 1) rank the features by coefficient value in a logistic regression ; 2) rank the features by "feature importance" from a random forest. These don't quite tell the same story, and I'm thinking that a feature that might be "unimportant" in a linear model could be very discriminative in a non-linear model that can "understand" it.

Is that true in general? Or should "important" features (those that contribute most to a classification score) be the same across all types of models?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ You are at the edge of a very important idea: interaction. One-at-a-time testing completely ignores and excludes this. It is a critical part of statistical design of experiments (DoE). In DoE you will see that the presumed model very strongly drives how you treat the variables, and that a weaker model can easily miss significant higher order interactions. Even the mighty random forest is imperfect in its measure of importance, though it accounts for nonlinearity and interactions. There are no silver bullets, but you will find there are different calibers. $\endgroup$
    – EngrStudent
    Commented Aug 24, 2020 at 18:05

2 Answers 2

Reset to default
4
$\begingroup$

When it comes to feature importance I always go with a model-agnostic measure, as you well mention if you have two different models, they will interpret importance in different terms (Linear models as the coefficient and Tree-based models as the information gain/impurity decrease on each feature.

So you already mention one measure that does not depends on the model, rather on the metric you are interested in; Permutation importance does not care about what model you are using, but the impact that a feature has on the global performance.

This reference might give you a better idea of the advantages of using permutation importance over tree-based models importance Permutation Importance vs Random Forest Feature Importance

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Hmm, yes, but - in the end, you will implement a definite model to e.g. detect cancer - why not look at feature importance under that model, rather than a model agnostic feature importance that might actually not give you the best insights for the model you implement? $\endgroup$
    – Frank
    Commented Aug 25, 2020 at 0:38
  • 2
    $\begingroup$ I think those are separate task, with model agnostic you are making feature selection whereas If you want to get the best insights for the model you implement you are talking about model explainability (suitable tools for such task are among others SHAP values, partial dependence plots). If still, you want to use the coefficients of the model for feature selection you will encounter what you already have, different models might tell different conclusions, but at the end is up to you $\endgroup$
    – Multivac
    Commented Aug 25, 2020 at 19:48
3
$\begingroup$

Your intuition so far is correct. Feature importance does not extend across models. The feature score for an xgboost model might be irrelevant and a wrong assumption for trsining another model. There is no perfect way to define important features. It does require some prior knowledge about the data in general.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.