I have a dataset consisting of around 30'000 data points and 3 classes. The classes are imbalanced (around 5'000 in class 1, 10'000 in class 2 and 15'000 in class 3). I'm building a convolutional neural network model for classification of the data. For evaluation I'm looking at the AUC and ROC curves. Because I have three classes I have to either use micro- or macro-average.
To calculate the micro- and macro-averaged AUC and ROC curve, I use the approach described here: https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html The micro-averaged AUC / ROC is calculated by considering each element of the label indicator matrix as a binary prediction and the macro-averaged AUC / ROC is calculated by calculating metrics for each label, and find their unweighted mean. In my case micro-averaged AUC is usually higher than macro-averaged AUC.
If we look at the
sklearn.metrics.roc_auc_score method it is written for
This does not take label imbalance into account.
I'm not sure if for micro-average, they use the same approach as it is described in the link above.
Is it better to use for dataset with class imbalance micro-average or macro-average? That means which metric is not affected by class imbalance? In my case micro-averaged AUC (0.85) is higher than macro-averaged AUC (0.79). When I look at the confusion matrix, the majority class is very well predicted (because the network probably learns to predict the majority class) but the minority classes are poorly predicted (almost as many false negatives as true positives). So, overall the AUC should not be that high I think.