How to get sensitivity and specificity for multi-class classification for each fold of cross validation?

Question

I am working on a multi-class classification consisting of 4 classes. I am applying 5-fold cross-validation on it and would like to get the sensitivity (recall) and specificity score for each of those folds.

I found out that using cross_validate function, I can provide it with a list of scoring parameters for each fold.

    scoring = {'accuracy' : make_scorer(accuracy), 
               'precision' : make_scorer(precision_score),
               'recall' : make_scorer(recall_score), 
               'f1_score' : make_scorer(f1_score)}

    cross_validate(neural_network, data, y, cv=5,scoring=scoring)

However, this generates an error because these functions (except accuracy) are only for binary classification and not for multi-class.

So, I decided to make my own functions for sensitivity score and specificity score that returns the mean of the 4 individual values (1 for each class). I return the mean of them, and not the individual 4 values because scorer functions that return multiple values are not permitted. That is fine with me though as I want their mean only.

Here's what I tried:

    def sensitivity(y_true,y_pred):
        cm=confusion_matrix(y_true, y_pred)
        FP = cm.sum(axis=0) - np.diag(cm)  
        FN = cm.sum(axis=1) - np.diag(cm)
        TP = np.diag(cm)
        TN = cm.sum() - (FP + FN + TP)
        Sensitivity = TP/(TP+FN)    
        return np.mean(Sensitivity)

    def specificity(y_true,y_pred):
        cm=confusion_matrix(y_true, y_pred)
        FP = cm.sum(axis=0) - np.diag(cm)  
        FN = cm.sum(axis=1) - np.diag(cm)
        TP = np.diag(cm)
        TN = cm.sum() - (FP + FN + TP)
        Specificity = TN/(TN+FP)    
        return np.mean(Specificity)


    scoring = {'sensitivity' : make_scorer(sensitivity),
               'specificity' : make_scorer(specificity)}

    cross_validate(neural_network, data, y, cv=5,scoring=scoring)

But it still is throwing the same error:

ValueError: Classification metrics can't handle a mix of multilabel-indicator and multiclass targets

I don't know what's not working here. I just want the mean of sensitivity for each class and mean of specificity for each class, for each of the 5 folds.

What is wrong with my approach and also is there a simpler way to do this ?

Answer 1

I think this error comes from confusion_matrix(), here we have three "types_of_target": multiclass, multilabel-indicator, continuous-multioutput.

For example, np.array([1, 0, 2]) is multiclass, it's one-hot-encoding np.array([[0,1,0],[1,0,0],[0,0,1]]) is multilabel-indicator, what we predict np.array([0.3,0.4,0.3],[0.7,0.2,0.1],[0.1,0.1,0.8]) is continuous-multioutput.

The input of confusion_matrix must be of type "multiclass".

I think you can try

confusion_matrix(y_true.argmax(axis=1),np.rint(y_pred).argmax(axis=1))

by converting y_true from multilabel-indicator to multiclass, and y_pred from probs(continuous-multioutput) to one-hot(multilabel-indicator) then multiclass.

Answer 2

Quick and dirty solution: (given that you dont want to do it "by hand")

Just use classification report, then just select metrics you want with averaging you want.