Is it possible to combine two confusion matrices?

Question

Assume I have two different algorithms that tests whether a given image contains a goat or not. I apply these two algorithms to two different datasets and obtain two confusion matrices.

Now I want to somehow combine these two algorithms into a third one as follows: Given an image, I apply both algorithms and claim the image contains a goat if both algorithms guesses so. If even one of them guesses the image has no goat, I return NO.

Is it possible to combine the original two confusion matrices into a third one in a meaningful way? Note that if the original two algorithms run on the same dataset, I could just combine the result to get the confusion matrix for the third one. (I guess using Cohen's kappa or Scott's pi?) However, this is not the case.

One way I could think of is as follows: Say the first dataset contains 10 images and the second one contains 20 images. I could select a random 10 images from the second dataset, and make the assumption that the first dataset actually equals this random 10 images from the second dataset. Then I can combine the results. Would that be a meaningful test?

Answer 1

No, this cannot work: even if the two confusion matrices were obtained from the same dataset there is no way to check the condition "both algorithms predict positive on the same image" from only the confusion matrices.

Example: take for instance an instance which is TP for A: it could be either a TP or a FN for B. Same thing for a FP for A: it can be either a FP for B or a TN. And so on, basically there's no way to deduce the number of TP or any other category for the meta-algorithm in this way. And this is assuming the same dataset in the first place.

So the only way to achieve this would be:

1. for the two algorithms to have been applied on the same dataset,
2. and to have the actual predictions (not only the confusion matrices) with the id of the image, so that the meta-model predictions can be obtained by a logical AND between the positive predictions of the two algorithms.