Predict_proba on a binary classification problem

Question

I have a binary classification task on my hands, i have a bunch of people that i need to classify as being ones or zeros and then use predict_proba to estimate how confident my prediction was on the samples used for inference. My understanding is that predict_proba for most classification algorithms isn't accurate and needs to be calibrated. Is there a common approach to get objectively accurate class probabilities ? Algorithms names , techniques and some code if possible. Thanks!

Note : my classes are imbalanced 80/20.

Answer 1

sklearn provides us with two methods to calibrate a probabilistic classifier via their CalibratedClassifierCV class; one using Platt's scaling (sigmoid) and one using isotonic regression.

Another method is by using a Venn-Abers predictor, which is not implemented in sklearn, however you can find a custom implementation here.

As for imbalanced datasets, make sure to exploit the class_weights argument of an sklearn classifier, to increase the weight of under-represented classes.