Visualising feature selection results for multiple classifiers and feature subset sizes

Question

I am using information gain feature selection technique to get different features subset sizes for my dataset, like so:

fs1 = SelectKBest(score_func=mutual_info_classif, k=10)
fs1.fit(X_train, y_train)
X_train_fs1 = fs1.transform(X_train)
X_test_fs1 = fs1.transform(X_test)


fs2 = SelectKBest(score_func=mutual_info_classif, k=20)
fs2.fit(X_train, y_train)
X_train_fs2 = fs2.transform(X_train)
X_test_fs2 = fs2.transform(X_test)


fs3 = SelectKBest(score_func=mutual_info_classif, k=30)
fs3.fit(X_train, y_train)
X_train_fs3 = fs3.transform(X_train)
X_test_fs3 = fs3.transform(X_test)

I am then testing the performance of 4 different algorithms (Logistic Regression, SVM, AdaBoost and Decision Trees) using different subset sizes of feature selected features (subset 1 has k=10, so 10 features, subset 2 has 20 features, etc.). To evaluate the model's performance, I am calculating the Precision, Recall and AUC, like so:

def compareAlgorithms(X_train, y_train, score):
    # Compare Algorithms
    seed = 7

    # prepare models
    models = []
    models.append(('LR', LogisticRegression()))
    models.append(('SVM', SVC()))
    models.append(('Linear SVC', LinearSVC()))
    models.append(('ADABOOST', AdaBoostClassifier()))
    models.append(('DT', DecisionTreeClassifier()))


    # evaluate each model in turn
    results = []
    names = []
    scoring = score
    
    print(score, ":")
    
    for name, model in models:
        skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=seed)
        #kfold = model_selection.KFold(n_splits=5, random_state=seed)
        cv_results = model_selection.cross_val_score(model, X_train, y_train, cv=skf, scoring=scoring)
        results.append(cv_results)
        names.append(name)
        msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
        print(msg)
        
    return results, names

As I now have lots of results, I am trying to plot the results to better visualise which algorithm is performing better with which subset. I want to create plots similar to the ones I found in this article:

I have tried using matplotlib to do this, but have found it quite difficult seeing as I am trying to plot different classifiers on different features subset. I can (sort of) plot a line plot of the algorithms performance for one data subset using this function:

def plot(results,names, score):
    import matplotlib.pyplot as plt
    # plot for algorithm comparison
    fig = plt.figure()
    fig.suptitle(score)
    ax = fig.add_subplot(111)
    plt.plot(results)
    ax.set_xticklabels(names)
    plt.show()
   

Which results in this plot:

The problem with the above plot (beside the overlapping model names below which I am working on fixing) is that it is for one feature subset.

Can anyone help me do a plot like the one in the paper I have attached and maybe direct me to useful resources for someone just starting to learn about data visualisation?

Many thanks.

Answer 1

You will have to study and understand Matplotlib and its tweaks.

This code will do the basic work. You can extend it. Also, please go through the references that are added at the end.

import matplotlib.pyplot as plt

data = {'AUC':{'RF':[0.7,0.2,0.5,0.9,0.4], 'LR':[0.9,0.25,0.35,0.99,0.55], 'SVM':[0.3,0.5,0.8,0.6,0.7] } }
x = ['S1','S2','S3','S4','S5']

plt.plot(x, data['AUC']['RF'], marker='^', linestyle='solid') 
plt.plot(x,data['AUC']['LR'], marker='o', color='r',linestyle='dashed')  
plt.plot(x,data['AUC']['SVM'], marker='s', color='b',linestyle='dashdot')  

$\hspace{3cm}$

References -
Matplotlib Official tutorial
Python Data Science Handbook by Jake VanderPlas
Matplotlib Lines
Matplotlib Markers
Matplotlib Colors

Answer 2

Try making these modifications to your function, it might look better.

import matplotlib.pyplot as plt
# optional but I like this style
# plt.style.use("seaborn-whitegrid")

def plot(results,names, score):
    # boxplot algorithm comparison
    fig = plt.figure()
    fig.suptitle(score)
    ax = fig.add_subplot(111)
    ax.plot(results, label = names, marker = "o", linestyle = "--")
    ax.set_ylabel(score)
    ax.legend(loc = "best")
    plt.show()

Here I'm not sure of the type of both score and names. But if both are strings it will work as I suggest

Answer 3

If you restructure your data a bit it is relatively straight forward:

import matplotlib.pyplot as plt

data = {
    "LR": [0.6, 0.7, 0.8, 0.7],
    "SVM": [0.7, 0.6, 0.8, 0.5],
    "Linear SVC": [0.8, 0.5, 0.7, 0.6],
    "ADABOOST": [0.7, 0.8, 0.6, 0.7],
    "DT": [0.6, 0.8, 0.5, 0.7]
}
subsets = [3, 5, 10, 20]

for model in data:
    plt.plot(subsets, data[model])

You can add a legend using plt.legend, axis titles can be set using plt.xlabel/plt.ylabel.

Answer 4

Thank you for all the suggested answers, they helped. What I ended up doing is the following:

Firstly I changed my function that compares between the different models like so:

def compareAlgorithmsFeatureSelection(X_train, y_train):
  
    # prepare configuration for cross validation test harness
    seed = 7

    # prepare models
    models = []
    models.append(('LR', LogisticRegression()))
    models.append(('SVM', SVC()))
    models.append(('Linear SVC', LinearSVC()))
    models.append(('ADABOOST', AdaBoostClassifier()))
    models.append(('DT', DecisionTreeClassifier()))

    
    names = []
    precision_results=[]
    recall_results=[]
    auc_results=[]
    
    NumOfKFeatures=[10,20,30,40,50]
    
    for name, model in models:
        names.append(name)
        precision_model_results = []
        recall_model_results = []
        auc_model_results = []
    
        for x in NumOfKFeatures:
            # get X_train and X_test after applying FS
            fs = SelectKBest(score_func=mutual_info_classif, k=x)
            fs.fit(X_train, y_train)
            X_train_fs = fs.transform(X_train)
            X_test_fs = fs.transform(X_test)

            # make splits for cross-validation
            skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=seed)

            # calculate scores
            precision_cv_results = model_selection.cross_val_score(model, X_train_fs,
                                                                       y_train, cv=skf, scoring='precision')
            recall_cv_results = model_selection.cross_val_score(model, X_train_fs,
                                                                       y_train, cv=skf, scoring='recall')
            auc_cv_results = model_selection.cross_val_score(model, X_train_fs,
                                                                       y_train, cv=skf, scoring='roc_auc')


            precision_model_results.append(precision_cv_results.mean()) 
            recall_model_results.append(recall_cv_results.mean())
            auc_model_results.append(auc_cv_results.mean())

        
        # append scores to final results list only after completed for an algorithm and all the feature subsets    
        precision_results.append(precision_model_results)
        print(precision_results)
        recall_results.append(recall_model_results)
        auc_results.append(auc_model_results)
        
        
    return precision_results, recall_results, auc_results, names

I then do the following to plot the score results for each feature subset using tips from answers provided by @Oxbowerce @10xAI and @Julio Jesus,:

import matplotlib.pyplot as plt
plt.style.use("seaborn-whitegrid")

subsets=[10,20,30,40,50]

fig = plt.figure()
fig.suptitle("Precision")
ax = fig.add_subplot(111)

for x in range(len(subsets)):
    ax.plot(subsets, precision_results[x], label = names[x], marker = "o", linestyle = "--")
    
ax.set_ylabel("Precision")
ax.set_xlabel("Feature Selection Subsets")
ax.legend(loc = "best")
plt.show()

To plot recall and auc I just replace precision_results with the relevant results list.