Is there a flexible way to get the original data indices from each cell of a confusion matrix?

Question

Let's say I have model A and model B, that I want to compare the performance of by inspecting a confusion matrix. They both produce a list of predictions, pred_A and pred_B, corresponding to a set list of ground truth values ground_truths. I can use something like sklearn.metrics.confusion_matrix in order to generate a confusion matrix for each of these models:

cm_A = confusion_matrix(ground_truths, pred_A) # e.g. [[5,1], [2,3]]
cm_B = confusion_matrix(ground_truths, pred_B) # e.g. [[6,0], [3,2]]

What I want to figure out is how to get the specific indices from each cell of the confusion matrix, corresponding to the ground truth class. This will let me investigate which specific data points are true-positives, false-positives, false negatives and true negatives, and let me search for patterns based on their input features. In the example above, are two of the false negatives the same false negatives in cm_A and cm_B? Or did all of the true positives switch with all of the false negatives between the two model predictions? If I knew their indices in the ground_truths list I could answer the question easily.

I can think of stupid/inefficient ways to do this, but I would love to find a solution that could scale to an arbitrary number of labels.

Answer 1

You can absolutely get this information, but not from the confusion matrices. You want to be comparing the prediction vectors themselves, not the confusion matrices, because as you've rightly identified, the confusion matrix dumps all the false postives / negatives into the same buckets (so that we can see how full each bucket is, also useful information).

You can do all sorts of comparisons of the prediction vectors to get the info you want (assuming you're using numpy arrays):

# Which examples did classifier A get wrong?
A_mistakes = np.invert(pred_A == ground_truths)

# Which examples did classifier B get wrong?
B_mistakes = np.invert(pred_B == ground_truths)

# Where did the classifiers make the same mistakes?
common_mistakes = A_mistakes == B_mistakes

# Where did the classifiers make a wrong prediction that the other didn't?
unique_mistakes = np.logical_xor(A_mistakes, B_mistakes)

You can then compare unique mistakes with A_mistakes or B_mistakes to trace them back to a classifier (or anything else you're interested in). You can also just np.sum the binary vectors to count the number of unique mistakes, common mistakes, etc.