1
$\begingroup$

I am doing a practice problem predicting a binary outcome. I have plotted an ROC curve and found the optimal threshold percentage to call future predicted observations a 1. I see that this threshold always matches the percentage of observations equal to 1 in my original data. Is there any conceptual explanation for this?

$\endgroup$
1
$\begingroup$

I see that this threshold always matches the percentage of observations equal to 1 in my original data. Is there any conceptual explanation for this?

Yes, although the fact it always matches exactly is probably a coincidence or maybe due to a small sample.

The training data contains a proportion $p$ of instances labelled 1. From the ROC plot you can see all the possible values for setting the threshold at a certain level and the resulting performance; for every possible level you can calculate the corresponding proportion $q$ of instances predicted as 1:

  • if $q$ is much lower than $p$, then the system predicts many 0s, so there are many false negative errors and that makes the recall lower. Precision is high in this case.
  • if $q$ is much higher than $p$, then the system predicts many 1s, so there are many false positive errors and that makes the precision lower. Recall is high in this case.

I assume that you optimize on the F1-score right? The fact that the F1-score is based on the product of the precision and recall means that both values need to be reasonably high, otherwise the F1-score drops. As seen above, having very different values for $p$ and $q$ will cause either the precision or recall to be low. Therefore the optimal F1-score is achieved when $q$ is close to $p$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.