Combining Classifiers with different Precision and Recall values

Question

Suppose I have two binary classifiers, A and B. Both are trained on the same set of data, and produce predictions on a different (but same for both classifiers) set of data. The precision for A is high and the recall is low, whereas the precision for B is low and the recall is high. Is there a way to combine these two models where I can get the precision from one and recall from the other, or possibly use the metrics from one to improve those of the other?

For example, let's say these are the metrics:

A

Precision: 0.91

Recall: 0.35

B

Precision: 0.44

Recall: 0.90

As these are binary classifiers, my labels are 1 and 0, and my class of interesting is 1 (so the metrics above are for predicting 1s).

Let's say model A predicts 10 1s, and model B predicts 70 1s.

It is safe for me to say that of the 10 1s that model A has predicted, 9 are true positives.

It is also safe for me to say that of the 70 1s that model B has predicted, 60% are false positives, but the rest are 90% of the true positives in the dataset.

My question is, is there some method for me to combine these results so that I can obtain all 50 true positives from the dataset?

Answer 1

You want to ensemble your two algorithms. The way to do that is to not just use e.g. sklearn's precision and recall metric functions, but to actually obtain probabilities for being positive (most models can output this with a simple function call), and take the average of those probabilities between your two (or more) binary classifiers. Then, test a range of probability thresholds beyond which you might label something positive, and evaluate the impact on recall and precision. This is a pretty common approach to obtaining better results through utilizing the strengths of different prediction algorithms.

Just some pythonic pseudo-code to demonstrate:

recall = []
precision = []
for threshold in [0.1, 0.2, 0.3, ..., 0.9]:
    df_of_mean_probabilities['label'] = df_of_mean_probabilities > threshold
    recall.append(TP/(TP + FN))
    precision.append(TP/(TP + FP))

Answer 2

It seems like your goal is to improve the performance of your model using the outputs of two existing models with different strengths. The specifics of how you might want to combine these two models will depend on your business problem

One approach is using your high-precision model (model A) to first identify a subset of instances that are very likely to be positive. You can then use your high-recall model (model B) to identify additional positive instances within the ones that model A is unsure about (the instances that model A did not classify as positive).

You could create a new composite model C that classifies an instance as positive if either A classifies it as positive (high-precision), or if both A is unsure and B classifies it as positive (high-recall). The assumption here is that the instances about which A is unsure could include many true positives, which B is more likely to correctly identify.

Another possible approach is stacking, where the predictions of multiple models are used as input features for a second-level model. You could train a model to take as input the prediction probabilities from models A and B and learn the best way to combine them to predict the true labels.oid overfitting.

Late on the answer but that’s the path I’d try first.

Answer 3

What you are actually looking for is Micro and Macro Precision and Recall.

If you directly have the precision and the recall values calculated, then all that's possible is only Macro Precision and Macro Recall.

Macro Formula goes as follows:

Macro_Precision = (Precision1 + Precision2) / 2

Macro_Recall = (Precision1 + Precision2) / 2

But if you have more granular details such as True Positives and False Positives, then you can compute even compute the Micro Precision and Micro Recall. Their formula goes as follows:

Micro_Precision = (TP1 + TP2) / (TP1 + TP2 + FP1 +FP2)

Micro_Recall = (TP1 + TP2) / (TP1 + TP2 + FN1 + FN2)

Micro and Macro have their own significance, you use Micro when you want insight in granular level, and this helps in cases where the Dataset is imbalanced.