How do I modify a Logistic Regression to target a specific point on the ROC curve?

Question

From a conceptual standpoint I understand the trade off involved with the ROC curve. You can increase the accuracy of true positive predictions but you will be taking on more false positives and vise versa.

I wondering how one would target a specific point on the curve for a Logistic Regression model? Would you just raise the probability threshold for what would constitute a 0 or a 1 in the regression? (Like shifting at what probability predictions start to get marked as one. ex: shifting the point predictions get marked 1 from 0.5 to 0.6)

I have a feeling it isn't that simple, but if it is how would you know which threshold to target to reach a specific point on the curve?

Answer 1

Yes, it is actually that simple :)

The ROC curve is made of all the points obtained by varying the classification threshold on the predicted score (usually a probability) above which instances gets predicted as positive. Increasing the thresholds causes less instances to be predicted as positive (higher precision, lower recall) and decreasing it causes the opposite.

Usually this is done by iterating across the unique predicted probabilities, calculating the performance (typically precision or recall) corresponding to using this value as threshold, and selecting the desired level of performance:

[edit: added detail below]

1. Take the unique values among all the probabilities for the instances in the test/validation set, then sort them
2. Then for each value considered as the threshold, calculate the performance on the full test/validation set.
3. pick the threshold which matches the required precision/recall

Actually a slightly more efficient version is to count the TP/FP/FN/TN instances directly on the instances sorted by probability, similarly to the short example in this answer.

Note: this is not specific to logistic regression, it applies to any binary classifier which provides scores/probabilities.