I am using information gain feature selection technique to get different features subset sizes for my dataset, like so:
fs1 = SelectKBest(score_func=mutual_info_classif, k=10)
fs1.fit(X_train, y_train)
X_train_fs1 = fs1.transform(X_train)
X_test_fs1 = fs1.transform(X_test)
fs2 = SelectKBest(score_func=mutual_info_classif, k=20)
fs2.fit(X_train, y_train)
X_train_fs2 = fs2.transform(X_train)
X_test_fs2 = fs2.transform(X_test)
fs3 = SelectKBest(score_func=mutual_info_classif, k=30)
fs3.fit(X_train, y_train)
X_train_fs3 = fs3.transform(X_train)
X_test_fs3 = fs3.transform(X_test)
I am then testing the performance of 4 different algorithms (Logistic Regression, SVM, AdaBoost and Decision Trees) using different subset sizes of feature selected features (subset 1 has k=10, so 10 features, subset 2 has 20 features, etc.). To evaluate the model's performance, I am calculating the Precision, Recall and AUC, like so:
def compareAlgorithms(X_train, y_train, score):
# Compare Algorithms
seed = 7
# prepare models
models = []
models.append(('LR', LogisticRegression()))
models.append(('SVM', SVC()))
models.append(('Linear SVC', LinearSVC()))
models.append(('ADABOOST', AdaBoostClassifier()))
models.append(('DT', DecisionTreeClassifier()))
# evaluate each model in turn
results = []
names = []
scoring = score
print(score, ":")
for name, model in models:
skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=seed)
#kfold = model_selection.KFold(n_splits=5, random_state=seed)
cv_results = model_selection.cross_val_score(model, X_train, y_train, cv=skf, scoring=scoring)
results.append(cv_results)
names.append(name)
msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
print(msg)
return results, names
As I now have lots of results, I am trying to plot the results to better visualise which algorithm is performing better with which subset. I want to create plots similar to the ones I found in this article:
I have tried using matplotlib to do this, but have found it quite difficult seeing as I am trying to plot different classifiers on different features subset. I can (sort of) plot a line plot of the algorithms performance for one data subset using this function:
def plot(results,names, score):
import matplotlib.pyplot as plt
# plot for algorithm comparison
fig = plt.figure()
fig.suptitle(score)
ax = fig.add_subplot(111)
plt.plot(results)
ax.set_xticklabels(names)
plt.show()
Which results in this plot:
The problem with the above plot (beside the overlapping model names below which I am working on fixing) is that it is for one feature subset.
Can anyone help me do a plot like the one in the paper I have attached and maybe direct me to useful resources for someone just starting to learn about data visualisation?
Many thanks.
k=10
). What do the multiple datapoints for the same algorithm represent? $\endgroup$ – Oxbowerce Jan 19 at 15:19