Comparing ML models to baselines

Question

When comparing ML models with baseline or "dummy" models, are there best practices for building and comparing baselines?

I'm doing a binary classification task where 40% of the samples are class_0 (untreated class), and the other 60% are class_1 (treated/positive class).

I have two baselines: baseline_0 predicts randomly, and baseline_1 predicts class_1 every time.

Because the metrics are calculated relative to class_1, when baseline_1 predicts class_1 on every sample, it ends up with perfect recall (1.0), which inflates the f1 score. Does this mean this baseline model is not good for comparing to my experimental models and I should use baseline_0 instead, or that baseline_1 is good but that the f1 score is not good for making these comparisons?

Answer 1

This depends on what you want to show. When working with metrics you shouldn't just take the value as is, but see what each metric are telling you. baseline_1 isn't better/worse than baseline_0 because it has a higher/lower value in metric X. Both baselines give an interesting perspective on a given dataset and if unsure I'd suggest keeping both.

A couple of notes:

• when saying baseline, I will refer to the two baseline strategies that you said in your post
• I will use the accuracy metric for examples but what I'm saying is true for any metric.

Why use baselines?

People usually tend to see accuracy (or other measures) as absolute values. E.g. accuracy=0.9? "very good", accuracy=0.3? "very bad". This isn't true however, as metrics are influenced by the number of classes and the proportion of samples between them.

However an accuracy of 0.3 in a classification task with 1000 classes is arguably much harder to achieve than an accuracy of 0.9 on a binary classification task (assuming class balance in both cases).

Here is were baselines come in. They can show how much better a model is than a dump classification strategy.

How baselines help?

Baselines help by putting a lower bound on your metrics. For example an accuracy of 0.55 on a binary classification task is slightly better than random, but the same accuracy on a 10-class setting is much better. Baselines help quantify that and tell you how much better you are than predicting random or the most common values.

What effect do baselines have?

Now on to why keep both baselines:

• The first baseline (i.e. random) helps show how metrics can be influenced by the number of classes has on the dataset.
• The second baseline (i.e. most common) helps show how metrics can be influenced by class imbalance.

How baselines actually help?

Let's you have two models, one with an accuracy of 0.92 and another with an accuracy of 0.93. How much better is the second model to the first? This depends on the value of your baseline. If you have a baseline accuracy of 0.5 then both models are relatively strong and the difference is not that significant. If you have a baseline of 0.9 then the models aren't as strong and an improvement of that magnitude is more significant.

Answer 2

Selecting the right metric is maybe the first point to find out and strongly depends on the real use case your are trying to apply the model to. Is it more important to prevent false negatives (e.g. in medical applications)? Or is it a balance between getting high true positives and true negatives ratio (e.g. not being too much alarmist in industrial applications)?
Once this is done, you can define what is a good baseline model for you, which could be:

• a completely effortless model, as the random classifier or the majority class predictor you mention
• a model that is already implemented in the product and you would like to improve

Whatever the type of baseline model you choose, your metric should come first after evaluating what is more important for your case and knowing which metric gives you more insight, so from the info you provide and not being too much unbalanced, I would say metrics like ROC-AUC is good for reaching a robust model, together with F-1 score with your aprox. 75% value as a good starting point without previous effort.

Answer 3

Does this mean this baseline model is not good for comparing to my experimental models and I should use baseline_0 instead, or that baseline_1 is good but that the f1 score is not good for making these comparisons?

By definition a majority classifier like baseline_1 is stronger than a random classifier like baseline_0 (unless the classes are uniformly distributed) since it always selects the majority class and therefore predicts more correct instances.

Here the recall is perfect for baseline_1 because the positive class is the majority class. In most tasks the minority class is chosen as the positive class, so this wouldn't happen. However that doesn't mean that the baseline is bad for the task, and it doesn't mean that F1-score is wrong either. With some tasks it can even happen that it's difficult for a "real" classifier to beat the baseline, but this doesn't necessarily mean that the "real" classifier is bad. A baseline classifier is just an indication of how a naive method performs.