Let's say I have model A and model B, that I want to compare the performance of by inspecting a confusion matrix. They both produce a list of predictions, pred_A
and pred_B
, corresponding to a set list of ground truth values ground_truths
. I can use something like sklearn.metrics.confusion_matrix
in order to generate a confusion matrix for each of these models:
cm_A = confusion_matrix(ground_truths, pred_A) # e.g. [[5,1], [2,3]]
cm_B = confusion_matrix(ground_truths, pred_B) # e.g. [[6,0], [3,2]]
What I want to figure out is how to get the specific indices from each cell of the confusion matrix, corresponding to the ground truth class. This will let me investigate which specific data points are true-positives, false-positives, false negatives and true negatives, and let me search for patterns based on their input features. In the example above, are two of the false negatives the same false negatives in cm_A
and cm_B
? Or did all of the true positives switch with all of the false negatives between the two model predictions? If I knew their indices in the ground_truths
list I could answer the question easily.
I can think of stupid/inefficient ways to do this, but I would love to find a solution that could scale to an arbitrary number of labels.