# How to perform Permutation Feature importance?

I am trying to perform feature selection. Currently with Tree based classifiers, even randomly generated column is ranking above some of my real columns. So I was reading about PFI. Can someone help me understand the steps to be followed?

1) Have my data in csv file

2) Fill in missing values/clean/prepare data

3) Split the data into train and test

4) Standardize train and test data

5) Select a model (let's say logistic regression)

6) Fit train data to logreg model

7) Predict using input test data

I have given the code below for my model

logreg=LogisticRegression() # step 5
logreg_cv.fit(X_train_std,y_train) # step 6
y_pred = logreg_cv.predict(X_test_std) # step 7


Would really be helpful if you could kind of explain in a way a noob can understand. I am new to ML, so it would really be helpful

Where does PFI fit in here? I see that we have eli5 package but couldn't find an example yet.

update - my code after suggested solution

logreg=LogisticRegression()
pi=PermutationImportance(logreg, cv=2).fit(X_train_std,y_train) # error is in this line.
rfe=RFECV(pi, step=1, n_jobs=-1, cv=5, scoring="auc").fit(X_train_std,y)
y_pred = rfe.predict(X_test_std)


Ok lets answer this with some basic steps:

Yo have done what you said until 5-th step after that you could do:

from sklearn.feature_selection import RFECV
from eli5.sklearn import PermutationImportance
pi=PermutationImportance(logreg, cv=2).fit(X_train,y)
selector=RFECV(pi, step=10, n_jobs=-1, cv=2, scoring="f1_macro").fit(X_train,y)


now make predictions

selector.predict(X_test)

or plot the most important features

import matplotlib.pyplot as plt plt.plot(selector.grid_scores_)

• this method can be used with any model. right? i mean instead of logreg, i can also use xgboost. Dec 17 '19 at 14:22
• Thats right, any object that is compactible with "estimator" in docs : eli5.readthedocs.io/en/latest/autodocs/… Dec 17 '19 at 14:24
• Hi, I keep encountering issue in the line i=PermutationImportance(logreg, cv=2).fit(X_train,y). that alueError: Input contains NaN, infinity or a value too large for dtype('float64').  Dec 18 '19 at 3:11
• Please find my code incorporating your suggestion above in the post. Please note that X_train_std is the standardized input data whereas y_train is the binary label column with 1s and 0s Dec 18 '19 at 3:17
• But X_train_std has no NA or inf. Not sure why it causes this error Dec 18 '19 at 3:18