It might be a stupid question, but I just realized that calling score function on logistic regression model shouldn't make any sense - as far as I know in sklearn we cannot specify threshold for model.predict called during evaluation, so we might be using suboptimal threshold and thus accuracy doesn't tell much about the model. I know how to extract probs from model and try different thresholds on my own, but asking if there is any point of using accuracy as a metric in cross-validation etc?? I would use roc_auc instead every time.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
The default threshold is 0.5, so it is computed on that basis. It is kind of a problem as it isn't well documented (not even on the logistic regression predict page) and a very common pitfall.
The two most common approach are either to use another metric (auc is a good exemple, proper scoring rule might be even better), the other approach is to buil a custom prediction function so as to incorporate your own threshold in your pipeline.
-
$\begingroup$ Thanks :) what do you mean by "proper scoring rule"? $\endgroup$ Mar 29 at 8:59
-
$\begingroup$ en.wikipedia.org/wiki/Scoring_rule typically... the idea is to use metrics that translate the probability being well calibrated $\endgroup$ Mar 29 at 9:11
-
$\begingroup$ @Icrmorin thanks again, for some reason I've never heard about proper scoring rules - I'll read about it and hopefully learn something important. $\endgroup$ Mar 29 at 9:16
-
$\begingroup$ I would say that for Logistic Regression threshold 0.5 is more or less justified, as typically Logistic Regression is used on top of other classifiers to make probabilities calibrated. $\endgroup$ Mar 29 at 12:06
-
$\begingroup$ Its far from being the main usecase. Ideally the theshold shoud be set depending on the cost of Type I and Type II errors. $\endgroup$ Mar 29 at 15:23