1
$\begingroup$

When representing calibration curves in a medical setting, usually they are represented the way they are in assays for lab analyses, where concentration analogue to "Fraction of Positives" is on the x-axis and "Predicted probability" on the y-axis. When looking at pure machine learning papers it seems to be the inverse. Those of you with expertise in the crossover of medicine and AI which way around do the axes go?

The plotted values are generated by taking my custom binary predictor model's probability output, binning that into 10 groups, for which it then checks the ground truth, and determines the fraction of positives for that group. This is the tool being used: https://scikit-learn.org/stable/modules/calibration.html#calibration

$\endgroup$

1 Answer 1

1
$\begingroup$

The argument for putting predicted probability on the horizontal axis seems to be:

  1. When we plot, we typically put the predicted value on the vertical axis, and the horizontal axis is for the value used to make the predictions.
  2. The known quantity is the prediction from your model. The unknown is the true probability to which that corresponds, which we calculate through various methods (e.g., rms::calibrate in R software).
  3. The predicted probability from a machine learning model might be totally unrelated to the reality of event occurrence, such as predicting $0.9$ when then event really only happens $60\%$ of the time. (This kind of overconfidence is often present in neural networks.)
  4. Therefore, we use the predicted values, plus a calibration step (if necessary) to get final predictions.

Perhaps think of it like precision in a classification problem: what is the probability of something being a positive case given that it is predicted to be positive. When you predict probabilities, you wonder about the probability of event occurrence given a model prediction of $p$.

The model prediction is an intermediate piece that is used as a feature in the calibration step and, thus, is on the horizontal axis. After all, you don’t need additional information about the true event occurrence rate to get the model to make a prediction when you already have the prediction!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.