1
$\begingroup$

I am working on a multi-class classification consisting of 4 classes. I am applying 5-fold cross-validation on it and would like to get the sensitivity (recall) and specificity score for each of those folds.

I found out that using cross_validate function, I can provide it with a list of scoring parameters for each fold.

    scoring = {'accuracy' : make_scorer(accuracy), 
               'precision' : make_scorer(precision_score),
               'recall' : make_scorer(recall_score), 
               'f1_score' : make_scorer(f1_score)}

    cross_validate(neural_network, data, y, cv=5,scoring=scoring)

However, this generates an error because these functions (except accuracy) are only for binary classification and not for multi-class.

So, I decided to make my own functions for sensitivity score and specificity score that returns the mean of the 4 individual values (1 for each class). I return the mean of them, and not the individual 4 values because scorer functions that return multiple values are not permitted. That is fine with me though as I want their mean only.

Here's what I tried:

    def sensitivity(y_true,y_pred):
        cm=confusion_matrix(y_true, y_pred)
        FP = cm.sum(axis=0) - np.diag(cm)  
        FN = cm.sum(axis=1) - np.diag(cm)
        TP = np.diag(cm)
        TN = cm.sum() - (FP + FN + TP)
        Sensitivity = TP/(TP+FN)    
        return np.mean(Sensitivity)

    def specificity(y_true,y_pred):
        cm=confusion_matrix(y_true, y_pred)
        FP = cm.sum(axis=0) - np.diag(cm)  
        FN = cm.sum(axis=1) - np.diag(cm)
        TP = np.diag(cm)
        TN = cm.sum() - (FP + FN + TP)
        Specificity = TN/(TN+FP)    
        return np.mean(Specificity)


    scoring = {'sensitivity' : make_scorer(sensitivity),
               'specificity' : make_scorer(specificity)}

    cross_validate(neural_network, data, y, cv=5,scoring=scoring)

But it still is throwing the same error:

ValueError: Classification metrics can't handle a mix of multilabel-indicator and multiclass targets

I don't know what's not working here. I just want the mean of sensitivity for each class and mean of specificity for each class, for each of the 5 folds.

What is wrong with my approach and also is there a simpler way to do this ?

$\endgroup$
1
$\begingroup$

I think this error comes from confusion_matrix(), here we have three "types_of_target": multiclass, multilabel-indicator, continuous-multioutput.

For example, np.array([1, 0, 2]) is multiclass, it's one-hot-encoding np.array([[0,1,0],[1,0,0],[0,0,1]]) is multilabel-indicator, what we predict np.array([0.3,0.4,0.3],[0.7,0.2,0.1],[0.1,0.1,0.8]) is continuous-multioutput.

The input of confusion_matrix must be of type "multiclass".

I think you can try

confusion_matrix(y_true.argmax(axis=1),np.rint(y_pred).argmax(axis=1))

by converting y_true from multilabel-indicator to multiclass, and y_pred from probs(continuous-multioutput) to one-hot(multilabel-indicator) then multiclass.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for pointing this out. The error was due to confusion_matrix only. The shape of y_true and y_pred was different. I just added this line at the start of my function to change the shape of y_true: y_true=y_true.argmax(axis=-1) Now, it's working perfectly. $\endgroup$ – Akshat Shrivastava Dec 28 '19 at 19:03
0
$\begingroup$

Quick and dirty solution: (given that you dont want to do it "by hand")

Just use classification report, then just select metrics you want with averaging you want.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.