0
$\begingroup$

I am designing a classifier for an Imbalanced Data set. I have a queries regarding choosing the threshold for a classifier, currently I am using mean of the predicted probabilities as the threshold and I am maximizing the recall on positive classes. Is this a correct way of choosing a threshold (I.e. mean of predicted probabilities) or I should try doing something like this book by using recall as a metric on KFold cross validation.

Any suggestion would be highly appreciable. Thanks :)

$\endgroup$
3
$\begingroup$

The threshold you choose depends on the specifics of the problem you are trying to solve. More specifically, it should be based on how you weigh false positives vs. false negatives, i.e. how bad each of these are relative to each other. You mention that you are trying to maximize recall on the positive class, but if that were true you would should just classify everything as a positive class, and get a recall of 1.0. Based on the domain you are working in, you should decide how much a false positive 'costs' vs. how much a false negative 'costs'. Once you decide this, you can find the threshold that minimizes the function

total cost = false negative count x FN cost + false positive count x FP cost

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.