0
$\begingroup$

I am trying to calibrate some classifiers to output more accurate probabilities. For this, I am using a sigmoid regression as implemented in sklearn.calibration.CalibratedClassifierCV with a 3-fold cross-validation and the ensembling method. However, after running tests on two data sets (both ~80 instances, ~15 features, binary target label), all of the four classifiers investigated are associated with a higher (worse) Brier score after calibration.

I have three questions:

a) Is the calibration just working bad in this case or is there a general procedure to improve calibration which I am missing out?

b) Is there something wrong with my implementation? (see code below)

c) If the code is okay, would you recommend deploying the uncalibrated model over the calibrated one since the Brier score is better?

Details:

  • Code
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import brier_score_loss
from sklearn.calibration import calibration_curve

calibrated_model = CalibratedClassifierCV(base_estimator=original_model,
                                          cv=3,
                                          ensemble=True,
                                          method="sigmoid")

calibrated_model.fit(features, labels)

original_probs = original_model.predict_proba(features)[:, 1]
calibrated_probs = calibrated_model.predict_proba(features)[:, 1]

fop_orig, mpv_orig = calibration_curve(labels.values, original_probs, n_bins=10, normalize=True)
fop_calib, mpv_calib = calibration_curve(labels.values, calibrated_probs, n_bins=10, normalize=True)

brier_loss_before = brier_score_loss(labels, original_probs, pos_label=np.max(labels))
brier_loss_after = brier_score_loss(labels, calibrated_probs, pos_label=np.max(labels))
  • Calibration curve examples

Logistic regression: Calibration curve example 1 Random forest: enter image description here

Thank you for answers in advance!

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ One possible reason for the calibration performing worse is that calibration requires a separate hold-out test set. I assume (but it's not clear from the posted code) that you use the same cv (cv=3) for the calibrated model and the original model. You could try fitting both uncalibrated and calibrated model on the same subset of data and using the remainder to calibrate. Also, consider using a cross-validate scheme with a random_seed, see e.g. stackoverflow.com/questions/73029007/… $\endgroup$
    – njp
    Feb 1 at 22:08
  • 1
    $\begingroup$ Thank you for your answer! In fact, the uncalibrated model was trained on the entire data set before plotting the calibration curve. This is likely to have introduced some overfitting which might be the reason for the reduced performance in comparison with the calibrated model. Seeding is another point I missed out on - Thank you! $\endgroup$
    – C.S.
    Feb 6 at 12:39

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.