1
$\begingroup$

I'm working on a multiclass classification problem, comparing results from SVM and Random Forest classificators. I would like to use gridsearchCV for hyperparameters tuning and find that AUC is the most used metrics for this kind of problem.

I know how to use other metrics of scoring like accuracy etc. but the default ROC_AUC only works for binary class. Is there a method to use AUC in gridsearchCV for multiclass problems?

$\endgroup$

3 Answers 3

0
$\begingroup$

sklearn's roc_auc_score actually does handle multiclass and multilabel problems, with its average and multiclass parameters. The default average='macro' is fine, though you should consider the alternative(s). But the default multiclass='raise' will need to be overridden. To use that in a GridSearchCV, you can curry the function, e.g.

import functools

multiclass_roc_auc = functools.partial(roc_auc_score, multiclass='ovr')
search = GridSearchCV(estimator=...,
                      eval=multiclass_roc_auc,
                      ...)
$\endgroup$
1
  • $\begingroup$ I fixed my problem by defining my multiclass_roc_auc function. Your solution is very helpful and slightly shorter, thanks a lot! $\endgroup$
    – okraw
    May 11, 2020 at 15:29
2
$\begingroup$

There are actually several flavors of AUC you can now use with multiclass evaluation:

  • 'roc_auc_ovo'
  • 'roc_auc_ovo_weighted'
  • 'roc_auc_ovr'
  • 'roc_auc_ovr_weighted'

These also work with BayesSearchCV from skopt and do not require creating your own scorer with functools.partial

$\endgroup$
1
$\begingroup$

Metrics are independent from ML algorithms, so it doesn't matter which algorithms did you use.

To calculate multiclass AUC you could use lib pRoc in R or use code this link(in Python).

Sources:

$\endgroup$
1
  • $\begingroup$ I mean the evaluation metrics that the GridsearchCV function uses to find best params $\endgroup$
    – okraw
    Apr 4, 2020 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.