0
$\begingroup$

I am aware of below approaches of feature selection

a) Feature Importance methods which are available in tree based models like Random Forest and Xgboost,GradientBoost etc.

b) statsmodel.logistic regression which in it's summary output provide us the results which contains whether variables are significant or not (P-value)

c) SelectKbest which uses ANOVA, Chi-square etc to compute the influence of input variable on target attribute

But unfortunately with methods b and c, it doesn't consider the feature interaction. Am I right? It works by considering each column to the target variable

Whereas with methods a it returns the ranking but we aren't sure about whether they are significant or not.

Is there anyway to know from Feature Importance whether the Features are significant or not? I understand features occurring in top 4-5 places could be significant but is there anyway to test/validate this?

Or is it like I pick each feature (out of say top 20 assuming they have a role) from feature importance result and do a SelectKbest test or statsmodel summary?

How can I know that the features that I select from Feature importance model are significant?

Improve this question
$\endgroup$
1
  • $\begingroup$ Hi, I was trying to add significance to the predictive features/add interpretive ability to my model. I am sharing this here as it is a related post. For which I did the below. can you help me with this post datascience.stackexchange.com/questions/65502/… $\endgroup$
    – The Great
    Commented Jan 2, 2020 at 5:24

2 Answers 2

Reset to default
2
$\begingroup$

1. univariate feature importance (that is c) in your list)

You are correct. Univariate statistics to estimate feature importance does not capture feature interactions. But they are fast and simple.

2. model-based feature importance (that are a) and b) in your list)

On the other hand model-based feature importance estimates can capture interactions as long as the model is capable of doing so (see "Introduction to Machine Learning with Python"; Mueller, Guido; 2017; p. 238/239). Which is not the case for linear regression.

For model-based feature importance estimates using trees there are ways to derive p-values. And at least R does have some implementations for that. Have a look at section "2.5 Importance testing procedures" in this paper The revival of the Gini importance?.

Improve this answer
$\endgroup$
4
  • $\begingroup$ no prob. Let me know if anything is still unclear. $\endgroup$
    – Jonathan
    Commented Dec 18, 2019 at 11:21
  • $\begingroup$ Is there any python package to do this? $\endgroup$
    – The Great
    Commented Dec 18, 2019 at 23:55
  • $\begingroup$ Can you also share the R package name that does this? or tutorial if any. I am new to ML and any step by step tutorial will help $\endgroup$
    – The Great
    Commented Dec 19, 2019 at 2:24
  • $\begingroup$ @SSMK Have a look at the vita package: cran.r-project.org/web/packages/vita/index.html . Also the authors of the paper I linked mention that the Ranger package has some implementations. $\endgroup$
    – Jonathan
    Commented Dec 19, 2019 at 9:01
1
$\begingroup$

Question 1. But unfortunately with methods b and c, it doesn't consider the feature interaction. Am I right? It works by considering each column to the target variable

None of the methods will take feature interactions into account See here

For a seperate discussion of feature interactions see here

Question 2. Is there anyway to know from Feature Importance whether the Features are significant or not? I understand features occurring in top 4-5 places could be significant but is there anyway to test/validate this

2 ways:

a) formally you can extend permutation importance to include p-values. Look at this paper

b) informally do cross-validation while performing some feature-selection

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.