Precision Vs Recall Curve analysis

Question

I have the following averaged 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛−𝑟𝑒𝑐𝑎𝑙𝑙 curves with 4 models. Which one is the best?

Answer 1

It depends.

The problem you are trying to solve decides which among the curves is the best

For example If you are trying to solve a problem that is like identifying cancer where the cost of false positive is extremely high. The first blue plot (model 3) is the best. However, you are compromising on recall or number of positive examples captured. It means that not all patients who have cancer will be identified as having cancer by model 3 but who ever the model says has cancer is most likely to have it amongst all the models plotted.

If your objective is just a simple marketing campaign where you would want most responders to be targetted but you are fine contacting people who are not likely to respond, model 2 which has the maximum recall.

When targetting a non responder is also expensive like a direct mail campaign, model 4 is the best because it balances both precision and recall.

Having said this response rate and sample size also need to be taken into account before arriving at which model is good. Because if accuracy of 94% and 87% are because of just 2 accurate samples (very low response) accuracy alone might not make sense.