# Visualising feature selection results for multiple classifiers and feature subset sizes

I am using information gain feature selection technique to get different features subset sizes for my dataset, like so:

fs1 = SelectKBest(score_func=mutual_info_classif, k=10)
fs1.fit(X_train, y_train)
X_train_fs1 = fs1.transform(X_train)
X_test_fs1 = fs1.transform(X_test)

fs2 = SelectKBest(score_func=mutual_info_classif, k=20)
fs2.fit(X_train, y_train)
X_train_fs2 = fs2.transform(X_train)
X_test_fs2 = fs2.transform(X_test)

fs3 = SelectKBest(score_func=mutual_info_classif, k=30)
fs3.fit(X_train, y_train)
X_train_fs3 = fs3.transform(X_train)
X_test_fs3 = fs3.transform(X_test)


I am then testing the performance of 4 different algorithms (Logistic Regression, SVM, AdaBoost and Decision Trees) using different subset sizes of feature selected features (subset 1 has k=10, so 10 features, subset 2 has 20 features, etc.). To evaluate the model's performance, I am calculating the Precision, Recall and AUC, like so:

def compareAlgorithms(X_train, y_train, score):
# Compare Algorithms
seed = 7

# prepare models
models = []
models.append(('LR', LogisticRegression()))
models.append(('SVM', SVC()))
models.append(('Linear SVC', LinearSVC()))
models.append(('DT', DecisionTreeClassifier()))

# evaluate each model in turn
results = []
names = []
scoring = score

print(score, ":")

for name, model in models:
skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=seed)
#kfold = model_selection.KFold(n_splits=5, random_state=seed)
cv_results = model_selection.cross_val_score(model, X_train, y_train, cv=skf, scoring=scoring)
results.append(cv_results)
names.append(name)
msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
print(msg)

return results, names


As I now have lots of results, I am trying to plot the results to better visualise which algorithm is performing better with which subset. I want to create plots similar to the ones I found in this article:

I have tried using matplotlib to do this, but have found it quite difficult seeing as I am trying to plot different classifiers on different features subset. I can (sort of) plot a line plot of the algorithms performance for one data subset using this function:

def plot(results,names, score):
import matplotlib.pyplot as plt
# plot for algorithm comparison
fig = plt.figure()
fig.suptitle(score)
plt.plot(results)
ax.set_xticklabels(names)
plt.show()



Which results in this plot:

The problem with the above plot (beside the overlapping model names below which I am working on fixing) is that it is for one feature subset.

Can anyone help me do a plot like the one in the paper I have attached and maybe direct me to useful resources for someone just starting to learn about data visualisation?

Many thanks.

• What code have you already written to try and create the plots? It should be relatively easy to plot different data series for the different classifiers. – Oxbowerce Jan 19 at 13:58
• @Oxbowerce I have added some of the code I have written. Appreciate any help you can give as I'm very new to data visualisation using python :) – sums22 Jan 19 at 14:30
• If I understand correctly the plot uses data for one feature subset (e.g. k=10). What do the multiple datapoints for the same algorithm represent? – Oxbowerce Jan 19 at 15:19
• Yes exactly, the 'results' variable contains the results for one of the subsets (e.g. k=10), for one score (e.g. precision). The multiple datapoints represent the results for the 5 different splits of the data in the cross validation. But I want to change this such that the x axis is the different data subsets and the y axis is the score value. Does that make sense? – sums22 Jan 19 at 15:39
• I think the answer from @10xAI shows a good start. You should probably look into restructuring the data from a list of lists to a dictionary. – Oxbowerce Jan 19 at 15:44

You will have to study and understand Matplotlib and its tweaks.

This code will do the basic work. You can extend it. Also, please go through the references that are added at the end.

import matplotlib.pyplot as plt

data = {'AUC':{'RF':[0.7,0.2,0.5,0.9,0.4], 'LR':[0.9,0.25,0.35,0.99,0.55], 'SVM':[0.3,0.5,0.8,0.6,0.7] } }
x = ['S1','S2','S3','S4','S5']

plt.plot(x, data['AUC']['RF'], marker='^', linestyle='solid')
plt.plot(x,data['AUC']['LR'], marker='o', color='r',linestyle='dashed')
plt.plot(x,data['AUC']['SVM'], marker='s', color='b',linestyle='dashdot')


$$\hspace{3cm}$$

Try making these modifications to your function, it might look better.

import matplotlib.pyplot as plt
# optional but I like this style
# plt.style.use("seaborn-whitegrid")

def plot(results,names, score):
# boxplot algorithm comparison
fig = plt.figure()
fig.suptitle(score)
ax.plot(results, label = names, marker = "o", linestyle = "--")
ax.set_ylabel(score)
ax.legend(loc = "best")
plt.show()


Here I'm not sure of the type of both score and names. But if both are strings it will work as I suggest

• Thanks you for your answer. Variable names contains a list of all the algorithm names that were used, I would need to iterate through the list to get a correct legend. – sums22 Jan 20 at 15:22

If you restructure your data a bit it is relatively straight forward:

import matplotlib.pyplot as plt

data = {
"LR": [0.6, 0.7, 0.8, 0.7],
"SVM": [0.7, 0.6, 0.8, 0.5],
"Linear SVC": [0.8, 0.5, 0.7, 0.6],
"ADABOOST": [0.7, 0.8, 0.6, 0.7],
"DT": [0.6, 0.8, 0.5, 0.7]
}
subsets = [3, 5, 10, 20]

for model in data:
plt.plot(subsets, data[model])


You can add a legend using plt.legend, axis titles can be set using plt.xlabel/plt.ylabel.

Thank you for all the suggested answers, they helped. What I ended up doing is the following:

Firstly I changed my function that compares between the different models like so:

def compareAlgorithmsFeatureSelection(X_train, y_train):

# prepare configuration for cross validation test harness
seed = 7

# prepare models
models = []
models.append(('LR', LogisticRegression()))
models.append(('SVM', SVC()))
models.append(('Linear SVC', LinearSVC()))
models.append(('DT', DecisionTreeClassifier()))

names = []
precision_results=[]
recall_results=[]
auc_results=[]

NumOfKFeatures=[10,20,30,40,50]

for name, model in models:
names.append(name)
precision_model_results = []
recall_model_results = []
auc_model_results = []

for x in NumOfKFeatures:
# get X_train and X_test after applying FS
fs = SelectKBest(score_func=mutual_info_classif, k=x)
fs.fit(X_train, y_train)
X_train_fs = fs.transform(X_train)
X_test_fs = fs.transform(X_test)

# make splits for cross-validation
skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=seed)

# calculate scores
precision_cv_results = model_selection.cross_val_score(model, X_train_fs,
y_train, cv=skf, scoring='precision')
recall_cv_results = model_selection.cross_val_score(model, X_train_fs,
y_train, cv=skf, scoring='recall')
auc_cv_results = model_selection.cross_val_score(model, X_train_fs,
y_train, cv=skf, scoring='roc_auc')

precision_model_results.append(precision_cv_results.mean())
recall_model_results.append(recall_cv_results.mean())
auc_model_results.append(auc_cv_results.mean())

# append scores to final results list only after completed for an algorithm and all the feature subsets
precision_results.append(precision_model_results)
print(precision_results)
recall_results.append(recall_model_results)
auc_results.append(auc_model_results)

return precision_results, recall_results, auc_results, names



I then do the following to plot the score results for each feature subset using tips from answers provided by @Oxbowerce @10xAI and @Julio Jesus,:


import matplotlib.pyplot as plt
plt.style.use("seaborn-whitegrid")

subsets=[10,20,30,40,50]

fig = plt.figure()
fig.suptitle("Precision")