7
$\begingroup$

I came across a new term called Calibration while reading about prediction models.

Can you please help me understand how different it is from Discrimination.

We build ML models to discriminate two/more classes from one another

But what does calibration mean and what does it mean to say that "The model has good discriminative power but poorly calibrated/calibrative power`?"

I thought we usually only look for separation only between 2 classes.

Can help me with this with a simple example please?

$\endgroup$
2
  • $\begingroup$ Calibration is based on range theory and discrimination is based on mathematics - quadratic thereom . A range is interval. Math is exact - the truth.. $\endgroup$ Mar 13, 2020 at 16:17
  • 1
    $\begingroup$ It seems that prediction models need a reinforcement by calibration of particular phenomenon e.g. occurrence of forest in the past. The postulated model needs a validation or affirmation- The model has good discriminative power but poorly calibrated/calibrative power`? ML - the maximum likelihood- Likelihood Ratio are frequently used in the Calibration endeavor. The Discriminant function (mathematical) is utilized to understand or ascertain say, growth of a present forest over the time. The preceding comment is somewhat technical and deviates from your query. $\endgroup$ Mar 14, 2020 at 2:32

4 Answers 4

5
$\begingroup$

Discrimination is the separation of the classes while calibration gives us scores based on risk of the population.

For example, there are 100 people that we’d like to predict a disease for and we know that only 3 out of 100 people have this disease. We get their probabilities from our model. Due to good predictability power, our model predicts probabilities between 0-0.05 for 70 people and 0.95-1 for 30 people. This is a good discrimination between classes. We now know that 30 people are at high risk considering only discrimination. But we also know that only 3 out of 100 people get the condition which is 3% prevalence. We use the 3% prevalence to calibrate our scores which will give the actual risk based on population of 100. That means, 0.95 x 0.03 = 0.0285 is their actual risk to the disease.

This is a very crude approach, there are advanced techniques like Kernels, Platt Scaling etc.,

$\endgroup$
1
  • $\begingroup$ Thanks for the response. Upvoted $\endgroup$
    – The Great
    Mar 14, 2020 at 1:26
4
$\begingroup$
  1. Calibration, agreement between observed and predicted risk, is more important in prognostic settings, because we would like to predict future risk of the target population, and the intercept (disease prevalence) is very important

  2. Discrimination, separating people with disease from without disease, is more important in diagnostic settings, because we want to diagnose the people as with/without disease using some test/factors under the predefined cutpoints. Good discrimination means that people with true disease will always have higher predicted risks than those people without disease. The intercept is not of interest.

Sometimes the model may be over-/under-estimate the risk (poor calibration), but it may still separate those with disease from without disease (good discrimination). Vice versa- the model may have good calibration, but cannot discriminate the cases from the control.

$\endgroup$
1
  • $\begingroup$ Thanks for the response. upvoted $\endgroup$
    – The Great
    Mar 14, 2020 at 1:26
1
$\begingroup$

Assume we want to predict an outcome for a number of people.

Discrimination: the model's ability to differentiate between those with the event from those without the event. Calibration: the agreement between the frequency of observed events with the predicted probabilities.

Example 1: a model correctly estimates that patient X has double the risk of an event as compared to patient Y. However, the model estimates that the probability of an event for patient X is 20% and for Y is 10%, while the true probabilities are 2% and 1%. The model has good discrimination but poor calibration.

Example 2: The model estimates for both X and Y a 1.5% probability of an event. The model is well calibrated but cannot tell you whether X or Y is more likely to have the event. The model is well calibrated but its discrimination is poor.

$\endgroup$
0
$\begingroup$

Assuming 50% prevalence (average event rate), here is an artificial simple example for illustration purpose only (in reality, the true probabilities are not known).

Comparing Calibration and Discrimination

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.