Multiclass imbalanced classification

Question

I have a dataset with the target variable having 3 classes. Value counts of Target variable are as follows:

Class A :26000
Class B: 5000
Class C: 500
Class C  prediction is most important from a business standpoint.

To date I have mostly done binary class (mostly 1:0.12) distribution of target class,wherein at the end, deciling of prediction probabilities is done, in decreasing order of probabilities starting with D1 as the highest probability and D2 the lowest.

Now with influx of a new class that too with so less population,I am perplexed on three aspects.

What algorithms are best? What metric should be used? How to get a deciling of probabilities in this case, if applicable?

Any leads/kernel/blog post would be really helpful.

Answer 1

You can weight the loss of each class with a suitable value which is inversely proportional to the class size. One example would be to use: (Total number of data points)/(number of data points of the class). For loss you can use the standard Binary Cross Entropy (BCE) loss.

As for the metric, I would advise you to use AUROC (Area Under the Receiver Operating Characteristics). You can generate an AUROC for each class to see the performance of the model on these individual class and average them to get an overall sense of your models performance.

Edit: I noticed you have put a tag on f1 score so I would like to point out that AUROC should be preferred as the metric as they more robust to class imbalance as compared to f1 scores.