# Binary classification problem with imbalanced dataset, how to compare to random classifier

We have a very imbalanced dataset (2% of class 1). To the best of our knowledge, there is no baseline in the literature to the problem we want to solve - so we thought of comparing our performance to a random classifier. We evaluate our model as a combination of precision and recall - we vary the threshold at which data points are classified as 1 and compute the rolling threshold and recall. We could use F1-score as well.

What would be an acceptable way to define a random predictor that we can compare to our model such that the comparison is as fair as possible?

You have $$98\%$$ in one class, right? This means that, knowing nothing about the data, you should be able to get $$98\%$$ of them right by guessing that majority class. If you get $$97\%$$ of them right, that sounds like an $$\text{A}$$ in school and thus a good model, but the model does worse than randomly guessing!

Better yet, compare using proper scoring rules like log loss (crossentropy) or Brier score, against a model that always predicts the prior probability of $$P(y=1) = 0.02$$. This is analogous to how $$R^2$$ works in linear regression, by always guessing the mean of the $$y$$ variable. In your case, the mean of the $$y$$ variable is the class ratio. If you can't beat the model that always guesses $$P(y=1) = 0.02$$, perhaps you have a poor model. (Specifics would depend on the misclassification costs, which you might or might not know.)

$$\text{Log Loss}\\ L(y, \hat y) = -\frac{1}{N}\sum_{i = 1}^N \bigg( y_i\log(\hat y_i) + (1 - y_i)\log(1 - \hat y_i) \bigg)$$ $$\text{Brier Score}\\ L(y, \hat y) = \frac{1}{N}\sum_{i = 1}^N \bigg(y_i - \hat y_i\bigg)^2$$

This assumes your $$y_i\in\{0, 1\}$$. If you use $$y_i\in\{-1. 1\}$$, you would have to modify the loss functions or change how you label your categories. The $$\hat y_i$$ values are probabilities. There are issues with the log loss if you predict a probability of $$0$$ or $$1$$. Some see this as an upside of log loss, while others see it as a downside.

This kind of evaluation of the probability outputs is why statisticians do not see class imbalance as an issue.

• Thank you very much for the answer. Aug 25 at 18:21

Since you are interested in different decision thresholds, your random model should produce scores. In that case, a reasonable base-line model assigns a score uniformly at random in $$[0,1]$$. Such a model will, at threshold $$t$$, have

\begin{align*} \operatorname{precision} &= \frac{2\%\cdot N\cdot (1-t)}{N(1-t)} = 0.02,\\[1em] \operatorname{recall} &= \frac{2\%\cdot N\cdot (1-t)}{2\%\cdot N} = 1-t. \end{align*}

(Perhaps a very simple model will serve as a better baseline.)