Discrimination vs Calibration - Machine Learning Models

Question

I came across a new term called Calibration while reading about prediction models.

Can you please help me understand how different it is from Discrimination.

We build ML models to discriminate two/more classes from one another

But what does calibration mean and what does it mean to say that "The model has good discriminative power but poorly calibrated/calibrative power`?"

I thought we usually only look for separation only between 2 classes.

Can help me with this with a simple example please?

Answer 1

Discrimination is the separation of the classes while calibration gives us scores based on risk of the population.

For example, there are 100 people that we’d like to predict a disease for and we know that only 3 out of 100 people have this disease. We get their probabilities from our model. Due to good predictability power, our model predicts probabilities between 0-0.05 for 70 people and 0.95-1 for 30 people. This is a good discrimination between classes. We now know that 30 people are at high risk considering only discrimination. But we also know that only 3 out of 100 people get the condition which is 3% prevalence. We use the 3% prevalence to calibrate our scores which will give the actual risk based on population of 100. That means, 0.95 x 0.03 = 0.0285 is their actual risk to the disease.

This is a very crude approach, there are advanced techniques like Kernels, Platt Scaling etc.,

Answer 2

1. Calibration, agreement between observed and predicted risk, is more important in prognostic settings, because we would like to predict future risk of the target population, and the intercept (disease prevalence) is very important

2. Discrimination, separating people with disease from without disease, is more important in diagnostic settings, because we want to diagnose the people as with/without disease using some test/factors under the predefined cutpoints. Good discrimination means that people with true disease will always have higher predicted risks than those people without disease. The intercept is not of interest.

Sometimes the model may be over-/under-estimate the risk (poor calibration), but it may still separate those with disease from without disease (good discrimination). Vice versa- the model may have good calibration, but cannot discriminate the cases from the control.

Answer 3

Assume we want to predict an outcome for a number of people.

Discrimination: the model's ability to differentiate between those with the event from those without the event. Calibration: the agreement between the frequency of observed events with the predicted probabilities.

Example 1: a model correctly estimates that patient X has double the risk of an event as compared to patient Y. However, the model estimates that the probability of an event for patient X is 20% and for Y is 10%, while the true probabilities are 2% and 1%. The model has good discrimination but poor calibration.

Example 2: The model estimates for both X and Y a 1.5% probability of an event. The model is well calibrated but cannot tell you whether X or Y is more likely to have the event. The model is well calibrated but its discrimination is poor.

Answer 4

Assuming 50% prevalence (average event rate), here is an artificial simple example for illustration purpose only (in reality, the true probabilities are not known).