i am training my xgboost model on an imbalanced binary classification problem. It is important to me to have well calibrated probabilities so i have chosen to optimize the brier score. I then plot the kde and reliability curve of my models where i try isotonic and platts.

enter image description here

e.g my grid search is:

gscv = GridSearchCV(pipeline, param_grid=params['xgboost'],scoring='neg_brier_score',
                    cv=kfolds.split(x_train, y_train),return_train_score=True)

the kde plot on the left corresponds to the uncalibrated probabilities , to me this looks good since the RED CURVE which is class 1 shows a bump towards lower end of curve , note the probabilites are the probability of default.

When i view the kde of the isotonic probabilities however it is not smooth:

enter image description here

my question is, are the above results satisfactory? I am not sure how i can improve this, for it to a) be smoother b) and be well calibrated.

  • $\begingroup$ What exactly do you mean by "smoother"? $\endgroup$ – Ben Reiniger Aug 19 '20 at 14:03
  • $\begingroup$ as in, why does my isotonic kde from above have bumps, it is not smooth, does this indicate overfitting? $\endgroup$ – Maths12 Aug 19 '20 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.