Suppose I have dataset labeled with two classes such as healthy and unhealthy and I applied feature selection (features importance)on dataset. How can I know that features are important to which class(to healthy or to unhealthy )?

  • $\begingroup$ In general feature importance in binary classification modeling helps is a measure of how much the feature help separating the two classes(not related to one class but to their difference). Please share how you preformed the feature selection. $\endgroup$ – yoav_aaa Mar 3 '19 at 9:23

Assuming we are talking about feature importance for decision tree algorithms here. You cannot really say. It only tells you how often a feature is used to split both classes apart.

If you want more insight in how your model makes decision you could look into SHAP and LIME. Both are methods that approximate your model and then tries to explain it. You can check out these two libraries in Python:



  • $\begingroup$ Thanks all for your answers . I perfmoed features secletion as in link below. machinelearningmastery.com/… $\endgroup$ – lona Apr 2 '19 at 11:03
  • $\begingroup$ @lona, glad you found a solution. If you want to be helpful you can always write what you did as an answer and mark it as correct. $\endgroup$ – Simon Larsson Apr 2 '19 at 11:30
  • $\begingroup$ Ok Simon ,Thanks. $\endgroup$ – lona May 2 '19 at 15:31

Something like this should get you going.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

df = pd.read_csv("https://rodeo-tutorials.s3.amazonaws.com/data/credit-data-trainingset.csv")

from sklearn.ensemble import RandomForestClassifier

features = np.array(['revolving_utilization_of_unsecured_lines',
                     'age', 'number_of_time30-59_days_past_due_not_worse',
                     'debt_ratio', 'monthly_income','number_of_open_credit_lines_and_loans', 
                     'number_of_times90_days_late', 'number_real_estate_loans_or_lines',
                     'number_of_time60-89_days_past_due_not_worse', 'number_of_dependents'])
clf = RandomForestClassifier()
clf.fit(df[features], df['serious_dlqin2yrs'])

# from the calculated importances, order them from most to least important
# and make a barplot so we can visualize what is/isn't important
importances = clf.feature_importances_
sorted_idx = np.argsort(importances)

padding = np.arange(len(features)) + 0.5
plt.barh(padding, importances[sorted_idx], align='center')
plt.yticks(padding, features[sorted_idx])
plt.xlabel("Relative Importance")
plt.title("Variable Importance")

enter image description here

  • $\begingroup$ Hi ASH, unfortunately, this does not answer the question of how important different predictors are for different target variables. As pointed out by @Simon Larsson, regular decision tree-based feature importance estimates do not address this. $\endgroup$ – Sammy Feb 23 '20 at 7:31
  • $\begingroup$ Oh, I didn't realize that. $\endgroup$ – ASH Feb 23 '20 at 13:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.