Assume I have two different algorithms that tests whether a given image contains a goat or not. I apply these two algorithms to two different datasets and obtain two confusion matrices.
Now I want to somehow combine these two algorithms into a third one as follows: Given an image, I apply both algorithms and claim the image contains a goat if both algorithms guesses so. If even one of them guesses the image has no goat, I return NO.
Is it possible to combine the original two confusion matrices into a third one in a meaningful way? Note that if the original two algorithms run on the same dataset, I could just combine the result to get the confusion matrix for the third one. (I guess using Cohen's kappa or Scott's pi?) However, this is not the case.
One way I could think of is as follows: Say the first dataset contains 10 images and the second one contains 20 images. I could select a random 10 images from the second dataset, and make the assumption that the first dataset actually equals this random 10 images from the second dataset. Then I can combine the results. Would that be a meaningful test?