Which scoring for GridSearchCV is best, when imbalanced multiclass dataset?

Question

I have an unbalanced multiclass dataset (GTSRB) and want to optimize the hyperparameters of an SVM through GridSearchCV. I know that accuracy is not suitable for scoring in this case. Which evaluation method for scoring would be most appropriate in this case?

At the moment I tend between the following: - f1_score (average='macro') - cohen_kappa_score

What are your experiences in such cases?

Answer 1

There is lots of metrics to measure performance of classifiers. The fundamental ones are based on the idea of:

• true positive (TP) — sample’s label is positive and it is classified as one
• true negative (TN) — sample’s label is negative and it is classified as one
• false positive (FP) — sample’s label is neg., but it is classified as positive
• false negative (FN) — sample’s label is pos., but it classified as negative

From what I have seen in white papers, F1-score is the most used metric that consider in imbalanced classification scenarios. But I also see ROC-AUC as a frequent used metric. As I mentioned, there is lots of metrics, but I strongly recommend you to keep these most used to provide to the others some standard sense of performance.

Answer 2

For the case of class imbalance, Precission-Recall AUC gives you a better insight than the usual ROC AUC, since it focuses on metrics dealing only with the minority class (assuming we label the minority class as positives, the event of interest, as usual). It is explained quite clearly in: https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-classification-in-python/

I think it is better to provide an AUC rather than a single F-score (meaning also a single precission and recall), since AUC means a metric for several decision thresholds...