1
$\begingroup$

I am using information gain feature selection technique to get different features subset sizes for my dataset, like so:

fs1 = SelectKBest(score_func=mutual_info_classif, k=10)
fs1.fit(X_train, y_train)
X_train_fs1 = fs1.transform(X_train)
X_test_fs1 = fs1.transform(X_test)


fs2 = SelectKBest(score_func=mutual_info_classif, k=20)
fs2.fit(X_train, y_train)
X_train_fs2 = fs2.transform(X_train)
X_test_fs2 = fs2.transform(X_test)


fs3 = SelectKBest(score_func=mutual_info_classif, k=30)
fs3.fit(X_train, y_train)
X_train_fs3 = fs3.transform(X_train)
X_test_fs3 = fs3.transform(X_test)

I am then testing the performance of 4 different algorithms (Logistic Regression, SVM, AdaBoost and Decision Trees) using different subset sizes of feature selected features (subset 1 has k=10, so 10 features, subset 2 has 20 features, etc.). To evaluate the model's performance, I am calculating the Precision, Recall and AUC, like so:

def compareAlgorithms(X_train, y_train, score):
    # Compare Algorithms
    seed = 7

    # prepare models
    models = []
    models.append(('LR', LogisticRegression()))
    models.append(('SVM', SVC()))
    models.append(('Linear SVC', LinearSVC()))
    models.append(('ADABOOST', AdaBoostClassifier()))
    models.append(('DT', DecisionTreeClassifier()))


    # evaluate each model in turn
    results = []
    names = []
    scoring = score
    
    print(score, ":")
    
    for name, model in models:
        skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=seed)
        #kfold = model_selection.KFold(n_splits=5, random_state=seed)
        cv_results = model_selection.cross_val_score(model, X_train, y_train, cv=skf, scoring=scoring)
        results.append(cv_results)
        names.append(name)
        msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
        print(msg)
        
    return results, names

As I now have lots of results, I am trying to plot the results to better visualise which algorithm is performing better with which subset. I want to create plots similar to the ones I found in this article:

enter image description here

I have tried using matplotlib to do this, but have found it quite difficult seeing as I am trying to plot different classifiers on different features subset. I can (sort of) plot a line plot of the algorithms performance for one data subset using this function:

def plot(results,names, score):
    import matplotlib.pyplot as plt
    # plot for algorithm comparison
    fig = plt.figure()
    fig.suptitle(score)
    ax = fig.add_subplot(111)
    plt.plot(results)
    ax.set_xticklabels(names)
    plt.show()
   

Which results in this plot:

enter image description here

The problem with the above plot (beside the overlapping model names below which I am working on fixing) is that it is for one feature subset.

Can anyone help me do a plot like the one in the paper I have attached and maybe direct me to useful resources for someone just starting to learn about data visualisation?

Many thanks.

$\endgroup$
7
  • $\begingroup$ What code have you already written to try and create the plots? It should be relatively easy to plot different data series for the different classifiers. $\endgroup$ – Oxbowerce Jan 19 at 13:58
  • $\begingroup$ @Oxbowerce I have added some of the code I have written. Appreciate any help you can give as I'm very new to data visualisation using python :) $\endgroup$ – sums22 Jan 19 at 14:30
  • $\begingroup$ If I understand correctly the plot uses data for one feature subset (e.g. k=10). What do the multiple datapoints for the same algorithm represent? $\endgroup$ – Oxbowerce Jan 19 at 15:19
  • $\begingroup$ Yes exactly, the 'results' variable contains the results for one of the subsets (e.g. k=10), for one score (e.g. precision). The multiple datapoints represent the results for the 5 different splits of the data in the cross validation. But I want to change this such that the x axis is the different data subsets and the y axis is the score value. Does that make sense? $\endgroup$ – sums22 Jan 19 at 15:39
  • $\begingroup$ I think the answer from @10xAI shows a good start. You should probably look into restructuring the data from a list of lists to a dictionary. $\endgroup$ – Oxbowerce Jan 19 at 15:44
1
$\begingroup$

You will have to study and understand Matplotlib and its tweaks.

This code will do the basic work. You can extend it. Also, please go through the references that are added at the end.

import matplotlib.pyplot as plt

data = {'AUC':{'RF':[0.7,0.2,0.5,0.9,0.4], 'LR':[0.9,0.25,0.35,0.99,0.55], 'SVM':[0.3,0.5,0.8,0.6,0.7] } }
x = ['S1','S2','S3','S4','S5']

plt.plot(x, data['AUC']['RF'], marker='^', linestyle='solid') 
plt.plot(x,data['AUC']['LR'], marker='o', color='r',linestyle='dashed')  
plt.plot(x,data['AUC']['SVM'], marker='s', color='b',linestyle='dashdot')  

$\hspace{3cm}$enter image description here

References -
Matplotlib Official tutorial
Python Data Science Handbook by Jake VanderPlas
Matplotlib Lines
Matplotlib Markers
Matplotlib Colors

$\endgroup$
1
$\begingroup$

Try making these modifications to your function, it might look better.

import matplotlib.pyplot as plt
# optional but I like this style
# plt.style.use("seaborn-whitegrid")

def plot(results,names, score):
    # boxplot algorithm comparison
    fig = plt.figure()
    fig.suptitle(score)
    ax = fig.add_subplot(111)
    ax.plot(results, label = names, marker = "o", linestyle = "--")
    ax.set_ylabel(score)
    ax.legend(loc = "best")
    plt.show()

Here I'm not sure of the type of both score and names. But if both are strings it will work as I suggest

$\endgroup$
1
  • $\begingroup$ Thanks you for your answer. Variable names contains a list of all the algorithm names that were used, I would need to iterate through the list to get a correct legend. $\endgroup$ – sums22 Jan 20 at 15:22
1
$\begingroup$

If you restructure your data a bit it is relatively straight forward:

import matplotlib.pyplot as plt

data = {
    "LR": [0.6, 0.7, 0.8, 0.7],
    "SVM": [0.7, 0.6, 0.8, 0.5],
    "Linear SVC": [0.8, 0.5, 0.7, 0.6],
    "ADABOOST": [0.7, 0.8, 0.6, 0.7],
    "DT": [0.6, 0.8, 0.5, 0.7]
}
subsets = [3, 5, 10, 20]

for model in data:
    plt.plot(subsets, data[model])

enter image description here

You can add a legend using plt.legend, axis titles can be set using plt.xlabel/plt.ylabel.

$\endgroup$
0
$\begingroup$

Thank you for all the suggested answers, they helped. What I ended up doing is the following:

Firstly I changed my function that compares between the different models like so:

def compareAlgorithmsFeatureSelection(X_train, y_train):
  
    # prepare configuration for cross validation test harness
    seed = 7

    # prepare models
    models = []
    models.append(('LR', LogisticRegression()))
    models.append(('SVM', SVC()))
    models.append(('Linear SVC', LinearSVC()))
    models.append(('ADABOOST', AdaBoostClassifier()))
    models.append(('DT', DecisionTreeClassifier()))

    
    names = []
    precision_results=[]
    recall_results=[]
    auc_results=[]
    
    NumOfKFeatures=[10,20,30,40,50]
    
    for name, model in models:
        names.append(name)
        precision_model_results = []
        recall_model_results = []
        auc_model_results = []
    
        for x in NumOfKFeatures:
            # get X_train and X_test after applying FS
            fs = SelectKBest(score_func=mutual_info_classif, k=x)
            fs.fit(X_train, y_train)
            X_train_fs = fs.transform(X_train)
            X_test_fs = fs.transform(X_test)

            # make splits for cross-validation
            skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=seed)

            # calculate scores
            precision_cv_results = model_selection.cross_val_score(model, X_train_fs,
                                                                       y_train, cv=skf, scoring='precision')
            recall_cv_results = model_selection.cross_val_score(model, X_train_fs,
                                                                       y_train, cv=skf, scoring='recall')
            auc_cv_results = model_selection.cross_val_score(model, X_train_fs,
                                                                       y_train, cv=skf, scoring='roc_auc')


            precision_model_results.append(precision_cv_results.mean()) 
            recall_model_results.append(recall_cv_results.mean())
            auc_model_results.append(auc_cv_results.mean())

        
        # append scores to final results list only after completed for an algorithm and all the feature subsets    
        precision_results.append(precision_model_results)
        print(precision_results)
        recall_results.append(recall_model_results)
        auc_results.append(auc_model_results)
        
        
    return precision_results, recall_results, auc_results, names

I then do the following to plot the score results for each feature subset using tips from answers provided by @Oxbowerce @10xAI and @Julio Jesus,:


import matplotlib.pyplot as plt
plt.style.use("seaborn-whitegrid")

subsets=[10,20,30,40,50]

fig = plt.figure()
fig.suptitle("Precision")
ax = fig.add_subplot(111)

for x in range(len(subsets)):
    ax.plot(subsets, precision_results[x], label = names[x], marker = "o", linestyle = "--")
    
ax.set_ylabel("Precision")
ax.set_xlabel("Feature Selection Subsets")
ax.legend(loc = "best")
plt.show()

enter image description here

To plot recall and auc I just replace precision_results with the relevant results list.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.