I'm testing out an XGBoost Classifier with the goal of using the probabilities it predicts in production. I know that tree based model probabilities are often not calibrated well so I decided to test both Sigmoid/Logistic + Isotonic regression to calibrate the probabilities, however, from both the calibration plot + log loss measurements as well as some spot checking it looks like the probabilities output after calibration are worse than the ones output by the regular classifier. Is there some way to understand why this is happening and if calibration is maybe the wrong thing to do in this scenario?

(ps. don't mind that the legend says "XGBoost Bayes", that is a typo but the underlying model is the XGBoost Classifier with sigmoid calibration - also the models for the log loss scores shown below are in the same order as the models calibration plot and are only scored on one test set)

enter image description here

the log loss score models are in the same order as the calibration

  • $\begingroup$ Additional Q: is it possible that a well tuned XGBoost model might output probabilities that are well-calibrated out of the box - and calibrating further might actually worsen performance? (obviously anything is possible, but i'm asking here if this has ever been witnessed in practice) $\endgroup$
    – Ted
    Commented Sep 19, 2023 at 16:51


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.