How to perform Permutation Feature importance?

Question

I am trying to perform feature selection. Currently with Tree based classifiers, even randomly generated column is ranking above some of my real columns. So I was reading about PFI. Can someone help me understand the steps to be followed?

1) Have my data in csv file

2) Fill in missing values/clean/prepare data

3) Split the data into train and test

4) Standardize train and test data

5) Select a model (let's say logistic regression)

6) Fit train data to logreg model

7) Predict using input test data

I have given the code below for my model

logreg=LogisticRegression() # step 5
logreg_cv.fit(X_train_std,y_train) # step 6
y_pred = logreg_cv.predict(X_test_std) # step 7

Would really be helpful if you could kind of explain in a way a noob can understand. I am new to ML, so it would really be helpful

Where does PFI fit in here? I see that we have eli5 package but couldn't find an example yet.

update - my code after suggested solution

logreg=LogisticRegression()
pi=PermutationImportance(logreg, cv=2).fit(X_train_std,y_train) # error is in this line. 
rfe=RFECV(pi, step=1, n_jobs=-1, cv=5, scoring="auc").fit(X_train_std,y)
y_pred = rfe.predict(X_test_std)

Answer 1

Ok lets answer this with some basic steps:

Yo have done what you said until 5-th step after that you could do:

from sklearn.feature_selection import RFECV
from eli5.sklearn import PermutationImportance
pi=PermutationImportance(logreg, cv=2).fit(X_train,y)
selector=RFECV(pi, step=10, n_jobs=-1, cv=2, scoring="f1_macro").fit(X_train,y)

now make predictions

selector.predict(X_test)

or plot the most important features

import matplotlib.pyplot as plt plt.plot(selector.grid_scores_)