1
$\begingroup$

I have a logistic regression model in Scikit-Learn doing a binary classification.

Looking at the Roc curve for the classifier I can see that it performs really well:

enter image description here

The AUC score is 0.99 which is great and looking at a probably distribution for the classifier, it has no problems separating the classes:

enter image description here

However, when I look at precision versus recall for different thresholds I see the following:

enter image description here

I know that it is possible to write a function in Scikit-Learn to use a custom threshold, depending upon business needs. I am wondering if this is something I need to do here?

I assume the default threshold is at 0 (I acknowledge that may be incorrect, but I am not sure) and looking at the Precision/Recall curve, the 'sweet spot' threshold to get the most out of Precision and Recall appears to be about 1.7. Can someone please let me know if that is correct or my process has errors in it, as I don't fully have a handle on best practice for this (yet). Thanks!

Improve this question
$\endgroup$
5
  • $\begingroup$ Not an answer, but you have some blue bars close to the green ones in the histogram, with a clear bimodal distribution. I would investigate these data points "manually"... it might be errors in labelling, or some signal you've missed. $\endgroup$
    – Calimo
    Jan 14, 2021 at 7:49
  • $\begingroup$ There's something that I don't understand: why is the range of values on the X axis different between the two graphs? I would expect that if the prediction is a probability (as it seems to be in the first graph), then the threshold should also be a probability in [0,1]. In general I don't think there's any default threshold, it's always calculated to optimize something (for instance F1-score). $\endgroup$
    – Erwan
    Jan 14, 2021 at 10:25
  • $\begingroup$ @Calimo the reason there is overlap between the two histograms is because those are false positives and false negatives. You're right, it is bi-modeal, but the reason why there is so much seperation is because the classifier is really good at separating the classes. $\endgroup$
    – Sandy Lee
    Jan 14, 2021 at 10:39
  • 1
    $\begingroup$ @Erwan, good spot. I see why you would say that. Threshold is Scikit-Learn parameter based on the decision function it uses to calculate the confidence of measurement and according to the documentation is 'the confidence score for a sample is the signed distance of that sample to the hyperplane'. You're right though, maybe probability would be better? $\endgroup$
    – Sandy Lee
    Jan 14, 2021 at 11:05
  • 1
    $\begingroup$ @Erwan. so I re-plotted this using probability and you are right, it makes more sense now. It seems to be showing that if I move my decision boundary to a probability of 0.8, then my classifier will have optimism precision/recall. However, my F1 score is already 0.98. I am going over Scikit-Learn's documentation now. $\endgroup$
    – Sandy Lee
    Jan 14, 2021 at 11:59

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.