I am using an XGBoost classifier to make risk predictions, and I see that even if it has very good binary classification results, the probability outputs are mainly under $0.05$ or over $0.95$ (like 60% of them).
I have tried calibration methods (from the sklearn
API) but it reduces the problem only slightly.
My dataset has 1800 training points and I test it on around 500 datapoints. It is quite balanced. I also use Bayesian optimisation to tune the hyperparameters of the model. There are 19 features for my model.
Does anyone know a solution to get more regularly distributed probabilities? Does the problem dwell in the fact I have too few datapoints? Should I set my hyperparameters differently? Do I have too many/few features?