I'm tackling a binary classification task using SVM implemented in python scikit-learn. Datasize is around 10,000 and the number of feature is 34.

After finding nice parameter set (using RandomizedSearchCV class), I evaluate the model by the cross validation. The result seems nice.

criteria_list = ["precision", "recall", "f1", "roc_auc"]
score_df = []
score_df2 = []
clf = svm.SVC(**random_search_clf.best_estimator_.get_params())
for crit in criteria_list:
    scores = cross_validation.cross_val_score(clf, X, y, cv=3, scoring=crit)
    score_df.append(["{} (±{})".format(np.round(np.mean(scores),3), np.round(np.std(scores),4)), scores])
    score_df2.append(["{} (±{})".format(np.round(np.mean(scores),3), np.round(np.std(scores),4))])

pd.DataFrame(np.transpose(score_df2), columns=criteria_list, index=["SVM"])

enter image description here

My question is whether it is possible to find out which feature is effective to classify the test data. I thought it's relating to sensitivity analysis, but good answer cannot be shown by googling "sensitivity analysis + svm" or "sensitivity analysis + scikit learn".


3 Answers 3


Example code of the nice suggestion from stmax above, with modification to use RandomForest and match the questions sample size and number of features, I hope that helps:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier

X, y = make_classification(n_samples=10000,

forest = RandomForestClassifier(n_estimators=250,

forest.fit(X, y)
importances = forest.feature_importances_
std = np.std([tree.feature_importances_ for tree in forest.estimators_],
indices = np.argsort(importances)[::-1]

# Print the feature ranking
print("Feature ranking:")

for f in range(X.shape[1]):
    print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))

# Plot the feature importances of the forest
plt.title("Feature importances")
plt.bar(range(X.shape[1]), importances[indices],
       color="g", yerr=std[indices], align="center")
plt.xticks(range(X.shape[1]), indices,rotation=60)
plt.xlim([-1, X.shape[1]])

Example feature importance plot


The effectiveness of features in your data depends on the "information gain" from that feature. The more the information gain, the better is the feature for your classification. I'm not exactly sure whether SVMs support such a technique to evaluate features, but you can look for the Decision Tree classification method. It calculates the entropy of the features, which then helps in calculating the information gain. From those calculations, you can easily find out which feature is effective to classify the test data.

  • 3
    $\begingroup$ I don't think SVMs do this. It is one of the criticisms of the approach listed in the wikipedia article that SVMs are too "black box" $\endgroup$ May 19, 2016 at 12:48
  • 2
    $\begingroup$ I have good experience with random forests for feature selection. Evaluate features with random forests, then feed the best ones to an SVM (or just stick to RF if model size doesn't matter :). $\endgroup$
    – stmax
    May 19, 2016 at 14:02
  • $\begingroup$ @stmax Thank you. Could you show how to evaluate features with random forests? $\endgroup$ May 20, 2016 at 6:13
  • $\begingroup$ Here's an example from the sklearn-docs: scikit-learn.org/stable/auto_examples/ensemble/… This one uses the ExtraTreesClassifier, but you can just change that to RandomForestClassifier.. both should give about the same results. $\endgroup$
    – stmax
    May 20, 2016 at 7:46

You are looking for model introspection capabilities, in other words, to make your model interpretable. There is quite a few techniques to do that (see this book by C. Molnar for some background), many of which are implemented in scikit-learn. I would start with permutation importance, which will estimate how much predictive power is lost by making a given feature meaningless.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.