2
$\begingroup$

I am running a CNN image multi-class classification model with Keras/Tensorflow and have established about a 90% overall accuracy with my best model trial. I have 10 unique classes I am trying to classify. However I want to present a PRC for the individual classes. I am trying to wrap my head around the intuition behind using thresholds for each class.

Since I utilize a SoftMax output layer activation I end up with a probability distribution for the 10 class label possibilities. If I am to create a PRC for this, I utilize different thresholds for each individual class and then classify the image as positive or negative depending on the probability.

Now in reality suppose I was using a threshold of 0.4 and my softmax activation gave me P(dog) = .41 and P(cat) = .42 and then small probabilities for the other classes. Thus with this threshold I am confirming my dog image as a dog even though the softmax probability was higher for cat.

Can someone explain to me how this makes sense in practice (if I were to deploy a classification model like this). My model would of course classify this image as a cat in practice since softmax chooses class with highest probability. What exactly is my PRC actually helping with when presenting evaluation metrics? Is it simply just showing how an individual class performs in the binary sense with a threshold for classification? In practice it wouldn't be using these thresholds for actual prediction.

Here is an example of what my PRC looks like for my validation set.

enter image description here

The code I used to produce it is utilizing one-vs-all methodology with sklearn

import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import cifar10
from keras.models import Model, load_model
from pathlib import Path
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_curve, auc, roc_auc_score, confusion_matrix, classification_report, precision_recall_curve, average_precision_score
from sklearn.preprocessing import label_binarize
from datetime import datetime
from time import time
from typing import Tuple
from itertools import cycle

def assess_pr_curve(model, xtrain: np.ndarray, ytrain: np.ndarray, xval: np.ndarray, yval: np.ndarray, xtest: np.ndarray, ytest: np.ndarray, 
                    save_plot_path: Path, n_classes: int, class_labels: list):
    """
    Get Precision and Recall and P/R Curve plots for Validation and Test data
    """

    feature_extractor = Model(inputs = model.inputs, outputs = model.get_layer('dense').output) # extract dense output layer (will be softmax probabilities)

    y_train_score = feature_extractor.predict(xtrain, batch_size = 64) # softmax probabilities for training data
    y_train_binary = label_binarize(ytrain, classes = [0,1,2,3,4,5,6,7,8,9]) # one hot encode train data

    y_val_score = feature_extractor.predict(xval, batch_size = 64) # softmax probability for validation data
    y_val_binary = label_binarize(yval, classes = [0,1,2,3,4,5,6,7,8,9]) # one hot encode validation data

    y_test_score = feature_extractor.predict(xtest, batch_size = 64) # one hot encoded softmax predictions
    y_test_binary = label_binarize(ytest, classes = [0,1,2,3,4,5,6,7,8,9]) # one hot encode the test data true labels

    # Precision-Recall Curves for train/val/test
    train_precision = dict()
    train_recall = dict()
    train_avg_precision = dict()

    val_precision = dict()
    val_recall = dict()
    val_avg_precision = dict()

    test_precision = dict()
    test_recall = dict()
    test_avg_precision = dict()

    for i in range(n_classes):
        train_precision[i], train_recall[i], _ = precision_recall_curve(y_train_binary[:, i], y_train_score[:, i])
        train_avg_precision[i] = average_precision_score(y_train_binary[:, i], y_train_score[:, i])
        val_precision[i], val_recall[i], _ = precision_recall_curve(y_val_binary[:, i], y_val_score[:, i])
        val_avg_precision[i] = average_precision_score(y_val_binary[:, i], y_val_score[:, i])
        test_precision[i], test_recall[i], _ = precision_recall_curve(y_test_binary[:, i], y_test_score[:, i])
        test_avg_precision[i] = average_precision_score(y_test_binary[:, i], y_test_score[:, i])

    colors = cycle(['blue', 'red', 'green', 'brown', 'purple', 'pink', 'orange', 'black', 'yellow', 'cyan'])

    # plot each class curve on single graph for multi-class one vs all classification
    for i, color, lbl in zip(range(n_classes), colors, class_labels):
        plt.plot(train_recall[i], train_precision[i], color = color, lw = 2,
        label = 'P/R Curve of class {0} (avg = {1:0.3f})'.format(lbl, train_avg_precision[i]))
    plt.hlines(0, xmin = -0.02, xmax = 1.0, linestyle = 'dashed')
    plt.xlim([-0.02, 1.03])
    plt.ylim([-0.03, 1.05])
    plt.xlabel('Recall')
    plt.ylabel('Precision')
    plt.title('Train P/R Curve for CIFAR-10 Multi-Class Data')
    plt.legend(loc = 'center left', prop = {'size': 6})
    fullpath = save_plot_path.joinpath('train_pr_curve.png')
    plt.savefig(fullpath)
    plt.close()


    # plot each class curve on single graph for multi-class one vs all classification
    for i, color, lbl in zip(range(n_classes), colors, class_labels):
        plt.plot(val_recall[i], val_precision[i], color = color, lw = 2,
        label = 'P/R Curve of class {0} (avg = {1:0.3f})'.format(lbl, val_avg_precision[i]))
    plt.hlines(0, xmin = -0.02, xmax = 1.0, linestyle = 'dashed')
    plt.xlim([-0.02, 1.03])
    plt.ylim([-0.03, 1.05])
    plt.xlabel('Recall')
    plt.ylabel('Precision')
    plt.title('Validation P/R Curve CIFAR-10 Multi-Class Data')
    plt.legend(loc = 'center left', prop = {'size': 6})
    fullpath = save_plot_path.joinpath('val_pr_curve.png')
    plt.savefig(fullpath)
    plt.close()

    # plot each class curve on single graph for multi-class one vs all classification
    for i, color, lbl in zip(range(n_classes), colors, class_labels):
        plt.plot(test_recall[i], test_precision[i], color = color, lw = 2,
        label = 'P/R Curve of class {0} (avg = {1:0.3f})'.format(lbl, test_avg_precision[i]))
    plt.hlines(0, xmin = -0.02, xmax = 1.0, linestyle = 'dashed')
    plt.xlim([-0.02, 1.03])
    plt.ylim([-0.03, 1.05])
    plt.xlabel('Recall')
    plt.ylabel('Precision')
    plt.title('Test P/R Curve for CIFAR-10 Multi-Class Data')
    plt.legend(loc = 'center left', prop = {'size': 6})
    fullpath = save_plot_path.joinpath('test_pr_curve.png')
    plt.savefig(fullpath)
    plt.close()
```
$\endgroup$
  • $\begingroup$ Can you provide a bit more info on how you built those curve (code ?). I suspect that they are 'one versus all' AUC curve, i.e. reformulating a multiclass problem into multiple binary ones. $\endgroup$ – lcrmorin Oct 18 '20 at 12:06
  • $\begingroup$ @Icrmorin Updated the post with the code. $\endgroup$ – Coldchain9 Oct 18 '20 at 16:17