# How to get sensitivity and specificity for multi-class classification for each fold of cross validation?

I am working on a multi-class classification consisting of 4 classes. I am applying 5-fold cross-validation on it and would like to get the sensitivity (recall) and specificity score for each of those folds.

I found out that using cross_validate function, I can provide it with a list of scoring parameters for each fold.

    scoring = {'accuracy' : make_scorer(accuracy),
'precision' : make_scorer(precision_score),
'recall' : make_scorer(recall_score),
'f1_score' : make_scorer(f1_score)}

cross_validate(neural_network, data, y, cv=5,scoring=scoring)


However, this generates an error because these functions (except accuracy) are only for binary classification and not for multi-class.

So, I decided to make my own functions for sensitivity score and specificity score that returns the mean of the 4 individual values (1 for each class). I return the mean of them, and not the individual 4 values because scorer functions that return multiple values are not permitted. That is fine with me though as I want their mean only.

Here's what I tried:

    def sensitivity(y_true,y_pred):
cm=confusion_matrix(y_true, y_pred)
FP = cm.sum(axis=0) - np.diag(cm)
FN = cm.sum(axis=1) - np.diag(cm)
TP = np.diag(cm)
TN = cm.sum() - (FP + FN + TP)
Sensitivity = TP/(TP+FN)
return np.mean(Sensitivity)

def specificity(y_true,y_pred):
cm=confusion_matrix(y_true, y_pred)
FP = cm.sum(axis=0) - np.diag(cm)
FN = cm.sum(axis=1) - np.diag(cm)
TP = np.diag(cm)
TN = cm.sum() - (FP + FN + TP)
Specificity = TN/(TN+FP)
return np.mean(Specificity)

scoring = {'sensitivity' : make_scorer(sensitivity),
'specificity' : make_scorer(specificity)}

cross_validate(neural_network, data, y, cv=5,scoring=scoring)


But it still is throwing the same error:

ValueError: Classification metrics can't handle a mix of multilabel-indicator and multiclass targets


I don't know what's not working here. I just want the mean of sensitivity for each class and mean of specificity for each class, for each of the 5 folds.

What is wrong with my approach and also is there a simpler way to do this ?

I think this error comes from confusion_matrix(), here we have three "types_of_target": multiclass, multilabel-indicator, continuous-multioutput.

For example, np.array([1, 0, 2]) is multiclass, it's one-hot-encoding np.array([[0,1,0],[1,0,0],[0,0,1]]) is multilabel-indicator, what we predict np.array([0.3,0.4,0.3],[0.7,0.2,0.1],[0.1,0.1,0.8]) is continuous-multioutput.

The input of confusion_matrix must be of type "multiclass".

I think you can try

confusion_matrix(y_true.argmax(axis=1),np.rint(y_pred).argmax(axis=1))


by converting y_true from multilabel-indicator to multiclass, and y_pred from probs(continuous-multioutput) to one-hot(multilabel-indicator) then multiclass.

• Thank you for pointing this out. The error was due to confusion_matrix only. The shape of y_true and y_pred was different. I just added this line at the start of my function to change the shape of y_true: y_true=y_true.argmax(axis=-1) Now, it's working perfectly. Dec 28, 2019 at 19:03

Quick and dirty solution: (given that you dont want to do it "by hand")

Just use classification report, then just select metrics you want with averaging you want.