1
$\begingroup$

I am using an XGBoost classifier to make risk predictions, and I see that even if it has very good binary classification results, the probability outputs are mainly under $0.05$ or over $0.95$ (like 60% of them).

I have tried calibration methods (from the sklearn API) but it reduces the problem only slightly.

My dataset has 1800 training points and I test it on around 500 datapoints. It is quite balanced. I also use Bayesian optimisation to tune the hyperparameters of the model. There are 19 features for my model.

Does anyone know a solution to get more regularly distributed probabilities? Does the problem dwell in the fact I have too few datapoints? Should I set my hyperparameters differently? Do I have too many/few features?

Improve this question
$\endgroup$
2
  • $\begingroup$ Where is the problem? Having big jumps in probabilities is, IMHO, related to a good discrimination of your algorithm : think about the logit function. Well, maybe that I misunderstand your question $\endgroup$
    – Michael Hooreman
    Oct 5, 2019 at 14:15
  • $\begingroup$ The thing is that sometimes the values are not so extreme, and sometimes they are. On what does the phenomenon depend? $\endgroup$
    – Ismalyt
    Oct 15, 2019 at 17:45

1 Answer 1

Reset to default
2
$\begingroup$

Since you're seeing probabilities concentrated near 0 and 1 (as is expected in gradient boosting), make sure you're not using Platt calibration; isotonic calibration is a better choice, or if you're willing to depart sklearn, beta calibration sounds promising. Keep in mind that all these calibration methods should use a separate training set. Also, have a look at calibration plots (which might show that Platt is unsuitable, and would also indicate whether the next paragraph is a possibility).

There's also a (slight) possibility that these scores are actually well-calibrated: e.g. if the data was generated in two clusters, and each cluster's points choose classes uniformly with probabilities close to 0 in one cluster and close to 1 in the other cluster, then "the right" answers are these probabilities, with the tree finding those clusters.

One approach to obtain "more regularly distributed probabilities" without actually calibrating them would be to make your model more conservative, using fewer trees and/or lower learning rate, more regularization, etc. but this risks hurting the model's performance.

Improve this answer
$\endgroup$
6
  • $\begingroup$ When you say calibration methods should use a different training dataset, you mean make the calibration adjustment not on the data used for model training? $\endgroup$
    – Ismalyt
    Oct 15, 2019 at 17:40
  • $\begingroup$ Regarding the size of my dataset, is isotonic calibration interesting? $\endgroup$
    – Ismalyt
    Oct 15, 2019 at 17:43
  • $\begingroup$ Yes, it is recommended not to use the model's train set for the calibration-model's training. Your small dataset that makes calibration potentially difficult; I'd worry in particular that the fit isotonic model would be too coarse. Try beta, but the data may still be too small for calibration to be very stable. (If AUC is your metric, you might be able to get away with using the test set for calibration-model training.) $\endgroup$
    – Ben Reiniger ♦
    Oct 16, 2019 at 14:23
  • $\begingroup$ @BenReiniger i have a dataset of around 300k and started with 1000 n estimators. i am trying to optimize the brier score through my grid search. it is really important for my problem that i have well calibrated probabilities. i am finding that isotonic calibration peforms better. However when i plot the kde of my class 0 and class 1 it's not very smooth. from a kde plot what would indicate well calibrated probabilities? $\endgroup$
    – Maths12
    Aug 13, 2020 at 13:18
  • $\begingroup$ @Maths12, that sounds like a separate question to post. $\endgroup$
    – Ben Reiniger ♦
    Aug 13, 2020 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.