i am training my xgboost model on an imbalanced binary classification problem. It is important to me to have well calibrated probabilities so i have chosen to optimize the brier score. I then plot the kde and reliability curve of my models where i try isotonic and platts.

enter image description here

e.g my grid search is:

gscv = GridSearchCV(pipeline, param_grid=params['xgboost'],scoring='neg_brier_score',
                    cv=kfolds.split(x_train, y_train),return_train_score=True)

the kde plot on the left corresponds to the uncalibrated probabilities , to me this looks good since the RED CURVE which is class 1 shows a bump towards lower end of curve , note the probabilites are the probability of default.

When i view the kde of the isotonic probabilities however it is not smooth:

enter image description here

my question is, are the above results satisfactory? I am not sure how i can improve this, for it to a) be smoother b) and be well calibrated.

  • $\begingroup$ What exactly do you mean by "smoother"? $\endgroup$
    – Ben Reiniger
    Commented Aug 19, 2020 at 14:03
  • $\begingroup$ as in, why does my isotonic kde from above have bumps, it is not smooth, does this indicate overfitting? $\endgroup$
    – Maths12
    Commented Aug 19, 2020 at 14:41


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.