12
$\begingroup$

1) Is it feasible to use the raw probabilities obtained from XGBoost, e.g. probabilities obtained within the range of 0.4-0.5, as a true representation of approximately 40%-50% chance of an event occurring? (assuming we have an accurate model)

2) The same question as in 1), but for other models that can output raw probabilities.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
14
$\begingroup$

It depends on the definition of accurate model, but in general the answer to your question 1) is No.

Regarding your second question (based on results in the paper of Niculescu-Mizil & Caruana linked below):

  • boosted trees and stumps - NO
  • Naive Bayes - NO
  • SVM - NO
  • bagged trees - YES
  • neural nets - YES

You can test whether it is the case for your particular model and dataset by looking at the so called reliability plot:

  • Create N bins based on the model output (e.g. 10-20)
  • Create a scatter plot with average model output for each bin along X axis and average true probability for each bin along Y axis

Ideally, your X-Y points should lie near the diagonal Y=X, otherwise the output of your classifier cannot be interpreted as a probability of an event.

However, not all is lost and if needed, one can try to modify (calibrate) the output of the model in a such way that it better reflects the observed probability. In order to assess whether the calibration exercise was successful one might look at the reliability plot based on the calibrated model output (instead of using raw model output).

Two most widely used techniques for the calibration of classifier outputs are Platt scaling and isotonic regression, see the links below.

Note, that it is not advisable to cailbrate the classifier using the training dataset (you might need to reserve a separate subset of your data for the purpose of calibration).

Some relevant links

Predicting Good Probabilities With Supervised Learning

CALIBRATING CLASSIFIER PROBABILTIES

Classifier calibration with Platt's scaling and isotonic regression

Improve this answer
$\endgroup$
3
  • $\begingroup$ After calibration, how would one be able to confirm that the probabilities obtained, are approximately representative ? Since predicting the likelihood that someone would develop a disease, 55% chance as compared with a 70% is a significant difference, and thus comes the importance of accurately predicting these probabilities. $\endgroup$
    – Gale
    Commented Mar 8, 2018 at 13:51
  • $\begingroup$ I've edited the answer: one would need another reliability plot (based on calibrated output). Note that you cannot do this assessment for every single person, only for the group of people who land in one bin (in terms of prediction). Also, according to my understanding, large number of bins could lead to poor results. $\endgroup$
    – aivanov
    Commented Mar 8, 2018 at 14:07
  • 2
    $\begingroup$ Old qustion, but for posterity: modern discriminative neural networks are definitively NOT calibrated in the sense that the OP posed (if they were, a big motivation for investigating generative networks would be redundant). See these papers: arxiv.org/pdf/1706.04599.pdf, papers.nips.cc/paper/… $\endgroup$
    – dant
    Commented Jul 1, 2020 at 23:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.