I am trying to calibrate some classifiers to output more accurate probabilities. For this, I am using a sigmoid regression as implemented in
sklearn.calibration.CalibratedClassifierCV with a 3-fold cross-validation and the ensembling method.
However, after running tests on two data sets (both ~80 instances, ~15 features, binary target label), all of the four classifiers investigated are associated with a higher (worse) Brier score after calibration.
I have three questions:
a) Is the calibration just working bad in this case or is there a general procedure to improve calibration which I am missing out?
b) Is there something wrong with my implementation? (see code below)
c) If the code is okay, would you recommend deploying the uncalibrated model over the calibrated one since the Brier score is better?
from sklearn.calibration import CalibratedClassifierCV from sklearn.metrics import brier_score_loss from sklearn.calibration import calibration_curve calibrated_model = CalibratedClassifierCV(base_estimator=original_model, cv=3, ensemble=True, method="sigmoid") calibrated_model.fit(features, labels) original_probs = original_model.predict_proba(features)[:, 1] calibrated_probs = calibrated_model.predict_proba(features)[:, 1] fop_orig, mpv_orig = calibration_curve(labels.values, original_probs, n_bins=10, normalize=True) fop_calib, mpv_calib = calibration_curve(labels.values, calibrated_probs, n_bins=10, normalize=True) brier_loss_before = brier_score_loss(labels, original_probs, pos_label=np.max(labels)) brier_loss_after = brier_score_loss(labels, calibrated_probs, pos_label=np.max(labels))
- Calibration curve examples
Thank you for answers in advance!