What does precision-recall curve and ROC curve tell us abouth threshold invariance

Question

Consider a binary classification problem.

Intuitively, a value for the area under the curve (for both curves) very close to 1, shows that the curve is almost L-shaped.

Thus, this means that the value on y axis stays rather consistent despite changes in threshold, and if we were to invert the axes, this would hold true for both values plotted.

Does this essentially mean that an L-shaped curve means that the model performs equally well (especially for the PR curve, since precision and recall are used to calculate F1 which is a pretty robust and widely used metric) for all classification thresholds? Or did I make some jump in my logic?

Answer 1

Does this essentially mean that an L-shaped curve means that the model performs equally well [...] for all classification thresholds

No it doesn't.

Let's consider a perfect predictor: all the positives are 1.0, all the negatives are 0.0. The curve is perfectly L shaped.

There are an infinity of thresholds between -Inf and 0.0 which will perform terribly, and classify all the negatives as positives. Similarly, all the thresholds between 1.0 and +Inf will incorrectly classify all the positive observations as negatives.

This is an extreme example, but you can easily extend it to cases where not all positives are 1.0 (for instance they are between 0.5 and 1.0) and not all negatives are 0.0 (for instance between 0.0 and 0.5), and see that there are a lot of possible thresholds that will missclassify observations. Here is a way to visualize:

You can see although the curve is L shaped, many classification points that missclassify a significant number of cases fall on the I and _ parts of the curve. Only the threshold 0.5 is a good classification threshold.