How to compute AUC in gridsearchSV (multiclass problem)

Question

I'm working on a multiclass classification problem, comparing results from SVM and Random Forest classificators. I would like to use gridsearchCV for hyperparameters tuning and find that AUC is the most used metrics for this kind of problem.

I know how to use other metrics of scoring like accuracy etc. but the default ROC_AUC only works for binary class. Is there a method to use AUC in gridsearchCV for multiclass problems?

Answer 1

sklearn's roc_auc_score actually does handle multiclass and multilabel problems, with its average and multiclass parameters. The default average='macro' is fine, though you should consider the alternative(s). But the default multiclass='raise' will need to be overridden. To use that in a GridSearchCV, you can curry the function, e.g.

import functools

multiclass_roc_auc = functools.partial(roc_auc_score, multiclass='ovr')
search = GridSearchCV(estimator=...,
                      eval=multiclass_roc_auc,
                      ...)

Answer 2

There are actually several flavors of AUC you can now use with multiclass evaluation:

• 'roc_auc_ovo'
• 'roc_auc_ovo_weighted'
• 'roc_auc_ovr'
• 'roc_auc_ovr_weighted'

These also work with BayesSearchCV from skopt and do not require creating your own scorer with functools.partial

Answer 3

Metrics are independent from ML algorithms, so it doesn't matter which algorithms did you use.

To calculate multiclass AUC you could use lib pRoc in R or use code this link(in Python).

Sources:

• https://web.expasy.org/pROC/
• https://medium.com/@plog397/auc-roc-curve-scoring-function-for-multi-class-classification-9822871a6659