3
$\begingroup$

I am working on multi-label classification problem, binary case. As a target variable there are five columns with 0-1 values.

For a model training I use scikit-multilearn library. Below is my code for training a model with Binary Relevance and RandomForest.

parameters = [
{
    'classifier': [RandomForestClassifier(random_state=42)],

    'classifier__n_estimators': [50, 100, 200],
    'classifier__max_features': ['auto', 'sqrt', 'log2'],
    'classifier__max_depth' : [4, 6, 8, 10, 12],
    'classifier__criterion' :['gini', 'entropy']}]

clf = RandomizedSearchCV(BinaryRelevance(), parameters, scoring='f1_weighted', cv=5, \
n_jobs=-1, verbose=10, random_state=10)
clf.fit(X_train, y_train)

How could I extract feature importance from the model? Should I take the best parameters from this model and then apply to data with each label separately using RandomForest from sklearn?

$\endgroup$
4
$\begingroup$

First, to directly answer your question, the easiest way to get Feature Importance using scikit learn is this, where model is the variable holding your classifier.

print(model.feature_importances_)

However, this method only exists on some of the Ensemble models, namely:

  • AdaBoostClassifier
  • AdaBoostRegressor
  • ExtraTreesClassifier
  • ExtraTreesRegressor
  • GradientBoostingClassifer
  • GradientBoostingRegressor
  • RandomForestClassifier
  • RandomForestRegressor
  • RandomTreesEmbedding

If you're wondering why, there's a fantastic free book online all about Interpretable Machine Learning. Here's an excerpt:

The easiest way to achieve interpretability is to use only a subset of algorithms that create interpretable models. Linear regression, logistic regression and the decision tree are commonly used interpretable models.

Once you've chosen the right model, beware of using feature importance! See here, it ranks random data very highly. The article proposes using Permutation Importance instead, as well as Drop-Column Importance.

They created a library called rfpimp for doing this, but here's a tutorial from scikit themselves on how to do both of those with just scikit-learn. I've pasted the permutation importance example from that tutorial below:

from sklearn.inspection import permutation_importance

result = permutation_importance(rf, X_test, y_test, n_repeats=10,
                                random_state=42, n_jobs=2)
sorted_idx = result.importances_mean.argsort()

fig, ax = plt.subplots()
ax.boxplot(result.importances[sorted_idx].T,
           vert=False, labels=X_test.columns[sorted_idx])
ax.set_title("Permutation Importances (test set)")
fig.tight_layout()
plt.show()
| improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Thank you for sharing the article. Itis interesting and I upvoted for that. However, BinaryRelevance from scikit-multilearn library doesn't work with feature_importances_ method. $\endgroup$ – Mirit Feb 21 at 13:58
  • 1
    $\begingroup$ @Mirit You are correct, I've edited my answer to include which models are supported as well as an explanation of why. Thanks! $\endgroup$ – Preston Badeer Feb 21 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.