2
$\begingroup$

I have a binary classification task on my hands, i have a bunch of people that i need to classify as being ones or zeros and then use predict_proba to estimate how confident my prediction was on the samples used for inference. My understanding is that predict_proba for most classification algorithms isn't accurate and needs to be calibrated. Is there a common approach to get objectively accurate class probabilities ? Algorithms names , techniques and some code if possible. Thanks!

Note : my classes are imbalanced 80/20.

$\endgroup$
1
$\begingroup$

sklearn provides us with two methods to calibrate a probabilistic classifier via their CalibratedClassifierCV class; one using Platt's scaling (sigmoid) and one using isotonic regression.

Another method is by using a Venn-Abers predictor, which is not implemented in sklearn, however you can find a custom implementation here.

As for imbalanced datasets, make sure to exploit the class_weights argument of an sklearn classifier, to increase the weight of under-represented classes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.