I am training a binary xgboost classifer with an imbalance of : 85% = 0 class and 14 % = class 1.

This was achieved after i took a random sample fromaround 11m to 1M.

When i calibrate i get the following:

enter image description here

It seems that using isotonic or sigmoid doesn't really improve the calbration much. Any idea how i can improve it?

    sig_clf = CalibratedClassifierCV(model, method="sigmoid", cv="prefit")
    iso_clf = CalibratedClassifierCV(model, method="isotonic", cv="prefit")

    sig_clf.fit(x_valid, y_valid)
    iso_clf.fit(x_valid, y_valid)
    prob_pos_sigmoid = sig_clf.predict_proba(x_test)[:, 1]
    prob_pos_iso = iso_clf.predict_proba(x_test)[:, 1]
    y_test_uncalibrated = model.predict_proba(x_test)[:, 1]

Above is what i use to plot the above graph. I have used 'prefit' since i calibrated after i have trained and fitted the model.

  • 1
    $\begingroup$ May not be able to improve the calibration. If the data is thin in the upper ends of the score range, then the calibration does not have a lot to go on. Can try to make a "better" base model with more features, feature engineering, etc. The usage of the model can discount the calibration accuracy at higher score levels. I have found the calibration drifts, depending on the model. When the input data drifts and the scores drift, the calibration has often drifted further for me. Might want to consider mapping to percentiles or other buckets for usage depending on the problem you are solving, $\endgroup$
    – Craig
    Jan 7 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.