How to get feature importance from RandomForest using scikit-multilearn library?

Question

I am working on multi-label classification problem, binary case. As a target variable there are five columns with 0-1 values.

For a model training I use scikit-multilearn library. Below is my code for training a model with Binary Relevance and RandomForest.

parameters = [
{
    'classifier': [RandomForestClassifier(random_state=42)],

    'classifier__n_estimators': [50, 100, 200],
    'classifier__max_features': ['auto', 'sqrt', 'log2'],
    'classifier__max_depth' : [4, 6, 8, 10, 12],
    'classifier__criterion' :['gini', 'entropy']}]

clf = RandomizedSearchCV(BinaryRelevance(), parameters, scoring='f1_weighted', cv=5, \
n_jobs=-1, verbose=10, random_state=10)
clf.fit(X_train, y_train)

How could I extract feature importance from the model? Should I take the best parameters from this model and then apply to data with each label separately using RandomForest from sklearn?

Answer 1

First, to directly answer your question, the easiest way to get Feature Importance using scikit learn is this, where model is the variable holding your classifier.

print(model.feature_importances_)

However, this method only exists on some of the Ensemble models, namely:

• AdaBoostClassifier
• AdaBoostRegressor
• ExtraTreesClassifier
• ExtraTreesRegressor
• GradientBoostingClassifer
• GradientBoostingRegressor
• RandomForestClassifier
• RandomForestRegressor
• RandomTreesEmbedding

If you're wondering why, there's a fantastic free book online all about Interpretable Machine Learning. Here's an excerpt:

The easiest way to achieve interpretability is to use only a subset of algorithms that create interpretable models. Linear regression, logistic regression and the decision tree are commonly used interpretable models.

Once you've chosen the right model, beware of using feature importance! See here, it ranks random data very highly. The article proposes using Permutation Importance instead, as well as Drop-Column Importance.

They created a library called rfpimp for doing this, but here's a tutorial from scikit themselves on how to do both of those with just scikit-learn. I've pasted the permutation importance example from that tutorial below:

from sklearn.inspection import permutation_importance

result = permutation_importance(rf, X_test, y_test, n_repeats=10,
                                random_state=42, n_jobs=2)
sorted_idx = result.importances_mean.argsort()

fig, ax = plt.subplots()
ax.boxplot(result.importances[sorted_idx].T,
           vert=False, labels=X_test.columns[sorted_idx])
ax.set_title("Permutation Importances (test set)")
fig.tight_layout()
plt.show()