How would you interpret the following ROC and PRC curves?
For example, I find it weird to understand that the precision actually increases at some point when recall increases as well. Is that even possible?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
My interpretation is that the data can be described as the following three groups, by decreasing order of the prediction to be positive:
For instance it might look like this:
<predicted probability> <true class> 1 P 0.9 P ... ... 0.4 P 0.38 N ... ... 0.14 N 0.11 P ... ... 0.05 P
This kind of configuration would explain the three stages which are clearly visible. There might be other ways this would happen, but I can't think of anything very different than this explanation (I could be wrong about that).
Note that this is why both precision and recall increase with this third group: since the threshold has already passed all the true negative instances (i.e. all the possible FP errors have been made), adding these true positive cases increases recall (more TP cases) without decreasing precision (no more FP). It's indeed very unusual, but not impossible.
What is interesting here is to investigate why this third group of positive instances are totally misclassified: they're not only confused with the negative (the common case), they are predicted as "more negative" than the true negative instances. This is very strange: it's as if there are two very distinct groups of positive instances that the classifier cannot predict on the same side of the positive/negative continuum. But it's encouraging that the classifier can quite well distinguish the three groups from each other, it means that it's probably something which can be fixed.