4
$\begingroup$

I came across a new term called Calibration while reading about prediction models.

Can you please help me understand how different it is from Discrimination.

We build ML models to discriminate two/more classes from one another

But what does calibration mean and what does it mean to say that "The model has good discriminative power but poorly calibrated/calibrative power`?"

I thought we usually only look for separation only between 2 classes.

Can help me with this with a simple example please?

| improve this question | | | | |
$\endgroup$
  • $\begingroup$ Calibration is based on range theory and discrimination is based on mathematics - quadratic thereom . A range is interval. Math is exact - the truth.. $\endgroup$ – Subhash C. Davar Mar 13 at 16:17
  • 1
    $\begingroup$ It seems that prediction models need a reinforcement by calibration of particular phenomenon e.g. occurrence of forest in the past. The postulated model needs a validation or affirmation- The model has good discriminative power but poorly calibrated/calibrative power`? ML - the maximum likelihood- Likelihood Ratio are frequently used in the Calibration endeavor. The Discriminant function (mathematical) is utilized to understand or ascertain say, growth of a present forest over the time. The preceding comment is somewhat technical and deviates from your query. $\endgroup$ – Subhash C. Davar Mar 14 at 2:32
2
$\begingroup$

Discrimination is the separation of the classes while calibration gives us scores based on risk of the population.

For example, there are 100 people that we’d like to predict a disease for and we know that only 3 out of 100 people have this disease. We get their probabilities from our model. Due to good predictability power, our model predicts probabilities between 0-0.05 for 70 people and 0.95-1 for 30 people. This is a good discrimination between classes. We now know that 30 people are at high risk considering only discrimination. But we also know that only 3 out of 100 people get the condition which is 3% prevalence. We use the 3% prevalence to calibrate our scores which will give the actual risk based on population of 100. That means, 0.95 x 0.03 = 0.0285 is their actual risk to the disease.

This is a very crude approach, there are advanced techniques like Kernels, Platt Scaling etc.,

| improve this answer | | | | |
$\endgroup$
  • $\begingroup$ Thanks for the response. Upvoted $\endgroup$ – The Great Mar 14 at 1:26
2
$\begingroup$
  1. Calibration, agreement between observed and predicted risk, is more important in prognostic settings, because we would like to predict future risk of the target population, and the intercept (disease prevalence) is very important

  2. Discrimination, separating people with disease from without disease, is more important in diagnostic settings, because we want to diagnose the people as with/without disease using some test/factors under the predefined cutpoints. Good discrimination means that people with true disease will always have higher predicted risks than those people without disease. The intercept is not of interest.

Sometimes the model may be over-/under-estimate the risk (poor calibration), but it may still separate those with disease from without disease (good discrimination). Vice versa- the model may have good calibration, but cannot discriminate the cases from the control.

| improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.