I am trying to perform feature selection. Currently with Tree based classifiers, even randomly generated column is ranking above some of my real columns. So I was reading about PFI. Can someone help me understand the steps to be followed?

1) Have my data in csv file

2) Fill in missing values/clean/prepare data

3) Split the data into train and test

4) Standardize train and test data

5) Select a model (let's say logistic regression)

6) Fit train data to logreg model

7) Predict using input test data

I have given the code below for my model

logreg=LogisticRegression() # step 5
logreg_cv.fit(X_train_std,y_train) # step 6
y_pred = logreg_cv.predict(X_test_std) # step 7

Would really be helpful if you could kind of explain in a way a noob can understand. I am new to ML, so it would really be helpful

Where does PFI fit in here? I see that we have eli5 package but couldn't find an example yet.

update - my code after suggested solution

pi=PermutationImportance(logreg, cv=2).fit(X_train_std,y_train) # error is in this line. 
rfe=RFECV(pi, step=1, n_jobs=-1, cv=5, scoring="auc").fit(X_train_std,y)
y_pred = rfe.predict(X_test_std)

1 Answer 1


Ok lets answer this with some basic steps:

Yo have done what you said until 5-th step after that you could do:

from sklearn.feature_selection import RFECV
from eli5.sklearn import PermutationImportance
pi=PermutationImportance(logreg, cv=2).fit(X_train,y)
selector=RFECV(pi, step=10, n_jobs=-1, cv=2, scoring="f1_macro").fit(X_train,y)

now make predictions


or plot the most important features

import matplotlib.pyplot as plt plt.plot(selector.grid_scores_)

  • $\begingroup$ this method can be used with any model. right? i mean instead of logreg, i can also use xgboost. $\endgroup$
    – The Great
    Dec 17, 2019 at 14:22
  • $\begingroup$ Thats right, any object that is compactible with "estimator" in docs : eli5.readthedocs.io/en/latest/autodocs/… $\endgroup$
    – Noah Weber
    Dec 17, 2019 at 14:24
  • 1
    $\begingroup$ Hi, I keep encountering issue in the line i=PermutationImportance(logreg, cv=2).fit(X_train,y). that alueError: Input contains NaN, infinity or a value too large for dtype('float64'). $\endgroup$
    – The Great
    Dec 18, 2019 at 3:11
  • $\begingroup$ Please find my code incorporating your suggestion above in the post. Please note that X_train_std is the standardized input data whereas y_train is the binary label column with 1s and 0s $\endgroup$
    – The Great
    Dec 18, 2019 at 3:17
  • $\begingroup$ But X_train_std has no NA or inf. Not sure why it causes this error $\endgroup$
    – The Great
    Dec 18, 2019 at 3:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.