# Probability calibration is worsening my model performance

I'm using RandomForest and XGBoost for binary classification, and my task is to predict probabilities for each class. Since tree-based models are bad with outputting usable probabilities, i imported the sklearn.calibration CalibratedClassifierCV, trained RF on 40k, then trained CCV with a separate 10k samples ( with cv="prefit" option ), my metric ( Area Under ROC ) is showing a huge drop in performance. Is it normal for probability calibration to alter the base estimator's behavior?

Edit : Since i'm minimizing logloss on my XGBClassifier, the output probabilities aren't that bad compared to RF's outputs.

• Are you reporting a drop in AUC on a third test set, going from the base RF model to the CalibratedClassifier? Aug 7 '19 at 18:51
• actually, the performance got worse for the dataset that i used for calibration. Aug 8 '19 at 8:20
• But still the drop is from the score of the RF on that calibration set? If so, are you able to share the (or a simplified) example? If not, then maybe it's just that the RF overfit the train set / the calibration set is not from the same distribution? Aug 8 '19 at 12:20