0
$\begingroup$

I am aware of below approaches of feature selection

a) Feature Importance methods which are available in tree based models like Random Forest and Xgboost,GradientBoost etc.

b) statsmodel.logistic regression which in it's summary output provide us the results which contains whether variables are significant or not (P-value)

c) SelectKbest which uses ANOVA, Chi-square etc to compute the influence of input variable on target attribute

But unfortunately with methods b and c, it doesn't consider the feature interaction. Am I right? It works by considering each column to the target variable

Whereas with methods a it returns the ranking but we aren't sure about whether they are significant or not.

Is there anyway to know from Feature Importance whether the Features are significant or not? I understand features occurring in top 4-5 places could be significant but is there anyway to test/validate this?

Or is it like I pick each feature (out of say top 20 assuming they have a role) from feature importance result and do a SelectKbest test or statsmodel summary?

How can I know that the features that I select from Feature importance model are significant?

$\endgroup$
  • $\begingroup$ Hi, I was trying to add significance to the predictive features/add interpretive ability to my model. I am sharing this here as it is a related post. For which I did the below. can you help me with this post datascience.stackexchange.com/questions/65502/… $\endgroup$ – The Great Jan 2 at 5:24
2
$\begingroup$

1. univariate feature importance (that is c) in your list)

You are correct. Univariate statistics to estimate feature importance does not capture feature interactions. But they are fast and simple.

2. model-based feature importance (that are a) and b) in your list)

On the other hand model-based feature importance estimates can capture interactions as long as the model is capable of doing so (see "Introduction to Machine Learning with Python"; Mueller, Guido; 2017; p. 238/239). Which is not the case for linear regression.

For model-based feature importance estimates using trees there are ways to derive p-values. And at least R does have some implementations for that. Have a look at section "2.5 Importance testing procedures" in this paper The revival of the Gini importance?.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Got it. thanks. $\endgroup$ – The Great Dec 18 '19 at 11:20
  • $\begingroup$ no prob. Let me know if anything is still unclear. $\endgroup$ – Sammy Dec 18 '19 at 11:21
  • $\begingroup$ Is there any python package to do this? $\endgroup$ – The Great Dec 18 '19 at 23:55
  • $\begingroup$ Can you also share the R package name that does this? or tutorial if any. I am new to ML and any step by step tutorial will help $\endgroup$ – The Great Dec 19 '19 at 2:24
  • $\begingroup$ @SSMK Have a look at the vita package: cran.r-project.org/web/packages/vita/index.html . Also the authors of the paper I linked mention that the Ranger package has some implementations. $\endgroup$ – Sammy Dec 19 '19 at 9:01
1
$\begingroup$

Question 1. But unfortunately with methods b and c, it doesn't consider the feature interaction. Am I right? It works by considering each column to the target variable

None of the methods will take feature interactions into account See here

For a seperate discussion of feature interactions see here

Question 2. Is there anyway to know from Feature Importance whether the Features are significant or not? I understand features occurring in top 4-5 places could be significant but is there anyway to test/validate this

2 ways:

a) formally you can extend permutation importance to include p-values. Look at this paper

b) informally do cross-validation while performing some feature-selection

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.