I'm using RandomForest and XGBoost for binary classification, and my task is to predict probabilities for each class. Since tree-based models are bad with outputting usable probabilities, i imported the sklearn.calibration CalibratedClassifierCV, trained RF on 40k, then trained CCV with a separate 10k samples ( with cv="prefit" option ), my metric ( Area Under ROC ) is showing a huge drop in performance. Is it normal for probability calibration to alter the base estimator's behavior?

Edit : Since i'm minimizing logloss on my XGBClassifier, the output probabilities aren't that bad compared to RF's outputs.

  • $\begingroup$ Are you reporting a drop in AUC on a third test set, going from the base RF model to the CalibratedClassifier? $\endgroup$ – Ben Reiniger Aug 7 '19 at 18:51
  • $\begingroup$ actually, the performance got worse for the dataset that i used for calibration. $\endgroup$ – Blenz Aug 8 '19 at 8:20
  • $\begingroup$ But still the drop is from the score of the RF on that calibration set? If so, are you able to share the (or a simplified) example? If not, then maybe it's just that the RF overfit the train set / the calibration set is not from the same distribution? $\endgroup$ – Ben Reiniger Aug 8 '19 at 12:20

The probability calibration is just stacking a logistic or isotonic regression on top of the base classifier. The default is logistic, and since the sigmoid is a strictly increasing function, the rank-ordering of samples will be unaffected, and so AUC should not change at all.

(With isotonic regression, it's actually piecewise-constant, so within a span where the function is constant, all samples will have their scores made equal, and so your ROC curve will become more coarse, which would affect the AUC; but these effects ought to be small, as long as the isotonic fit produces sufficiently many/short constant segments.)

Also, gradient boosting like XGBoost produces scores biased toward the extremes, not away from them like random forests, so the logistic calibration is unlikely to work out well.

| improve this answer | |
  • $\begingroup$ if not logistic calibration on tree-based models, then isotonic should be fine? or should i look for something else? $\endgroup$ – Blenz Aug 8 '19 at 8:21
  • $\begingroup$ It's specifically gradient boosted trees, not all tree models, where logistic/ sigmoid/Platt is ill-suited. Check a calibration plot; the GBM probably won't look sigmoidal. Isotonic should be fine, but check calibration afterward. $\endgroup$ – Ben Reiniger Aug 8 '19 at 12:13
  • $\begingroup$ btw, i'm using logistic loss as objective in my xgboost model, i checked the probabilities output by predict_proba, and i think i can sacrify the discrepancies of the probabilities which are not that static ,to be honest,to keep a good classification score on my metric. $\endgroup$ – Blenz Aug 8 '19 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.