0
$\begingroup$

It might be a stupid question, but I just realized that calling score function on logistic regression model shouldn't make any sense - as far as I know in sklearn we cannot specify threshold for model.predict called during evaluation, so we might be using suboptimal threshold and thus accuracy doesn't tell much about the model. I know how to extract probs from model and try different thresholds on my own, but asking if there is any point of using accuracy as a metric in cross-validation etc?? I would use roc_auc instead every time.

$\endgroup$

1 Answer 1

2
$\begingroup$

The default threshold is 0.5, so it is computed on that basis. It is kind of a problem as it isn't well documented (not even on the logistic regression predict page) and a very common pitfall.

The two most common approach are either to use another metric (auc is a good exemple, proper scoring rule might be even better), the other approach is to buil a custom prediction function so as to incorporate your own threshold in your pipeline.

$\endgroup$
5
  • $\begingroup$ Thanks :) what do you mean by "proper scoring rule"? $\endgroup$ Mar 29, 2023 at 8:59
  • $\begingroup$ en.wikipedia.org/wiki/Scoring_rule typically... the idea is to use metrics that translate the probability being well calibrated $\endgroup$ Mar 29, 2023 at 9:11
  • $\begingroup$ @Icrmorin thanks again, for some reason I've never heard about proper scoring rules - I'll read about it and hopefully learn something important. $\endgroup$ Mar 29, 2023 at 9:16
  • $\begingroup$ I would say that for Logistic Regression threshold 0.5 is more or less justified, as typically Logistic Regression is used on top of other classifiers to make probabilities calibrated. $\endgroup$ Mar 29, 2023 at 12:06
  • $\begingroup$ Its far from being the main usecase. Ideally the theshold shoud be set depending on the cost of Type I and Type II errors. $\endgroup$ Mar 29, 2023 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.