1
$\begingroup$

Let's say I have model A and model B, that I want to compare the performance of by inspecting a confusion matrix. They both produce a list of predictions, pred_A and pred_B, corresponding to a set list of ground truth values ground_truths. I can use something like sklearn.metrics.confusion_matrix in order to generate a confusion matrix for each of these models:

cm_A = confusion_matrix(ground_truths, pred_A) # e.g. [[5,1], [2,3]]
cm_B = confusion_matrix(ground_truths, pred_B) # e.g. [[6,0], [3,2]]

What I want to figure out is how to get the specific indices from each cell of the confusion matrix, corresponding to the ground truth class. This will let me investigate which specific data points are true-positives, false-positives, false negatives and true negatives, and let me search for patterns based on their input features. In the example above, are two of the false negatives the same false negatives in cm_A and cm_B? Or did all of the true positives switch with all of the false negatives between the two model predictions? If I knew their indices in the ground_truths list I could answer the question easily.

I can think of stupid/inefficient ways to do this, but I would love to find a solution that could scale to an arbitrary number of labels.

Improve this question
$\endgroup$
2
$\begingroup$

You can absolutely get this information, but not from the confusion matrices. You want to be comparing the prediction vectors themselves, not the confusion matrices, because as you've rightly identified, the confusion matrix dumps all the false postives / negatives into the same buckets (so that we can see how full each bucket is, also useful information).

You can do all sorts of comparisons of the prediction vectors to get the info you want (assuming you're using numpy arrays):

# Which examples did classifier A get wrong?
A_mistakes = np.invert(pred_A == ground_truths)

# Which examples did classifier B get wrong?
B_mistakes = np.invert(pred_B == ground_truths)

# Where did the classifiers make the same mistakes?
common_mistakes = A_mistakes == B_mistakes

# Where did the classifiers make a wrong prediction that the other didn't?
unique_mistakes = np.logical_xor(A_mistakes, B_mistakes)

You can then compare unique mistakes with A_mistakes or B_mistakes to trace them back to a classifier (or anything else you're interested in). You can also just np.sum the binary vectors to count the number of unique mistakes, common mistakes, etc.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks Matthew. The assumption of numpy arrays is a fine one. Is there a flexible way to distinguish between mistakes in a numpy-ish way? In the case of a binary classifier, this would be between FPs and FNs. But for three labels, the logical arithmetic gets a bit more complicated. $\endgroup$
    – Saul Aryeh Kohn
    Jan 7 at 20:50
  • $\begingroup$ I think I'd need to know more exactly what you're trying to do. Above, I'm looking at where the classifiers made mistakes on the same examples - if you want to know if they made the same mistakes (got them wrong in the same way) then you could just check equality for pred_A == pred_B where common_mistakes is true. You basically want a (ground-truth, predicted-label) tuple for each data point for classifier A and classifier B, and to look for patterns in the data matrix for which those pairs are equal? $\endgroup$
    – Matthew
    Jan 7 at 21:16
  • $\begingroup$ Yeah maybe I can clarify a little more. The impetus for the question is model comparison, but really concentrating on a single model will do the trick. If I 10 data points in my validation set, I'll be working with two integer arrays of 10 elements each: pred_A and ground_truths. For each element in pred_A, I would like to know which zone of the confusion matrix it landed in. For example, perhaps TPs = [0,1,5], where the integers in that list correspond to elements of ground_truths. For a binary classifier this is trivial, but for a multiclass one a solution is harder to find. $\endgroup$
    – Saul Aryeh Kohn
    Jan 7 at 21:24
  • $\begingroup$ The coordinates of the cell of the confusion matrix that a data point lands in are just (ground_truth_label, predicted_label), if the labels are integers $[0, C-1]$ for $C$ classes. You can get those indices in an $n$ by 2 array (assuming $n$ data points) with cm_indices = np.hstack((ground_truth.reshape(-1,1), pred_A.reshape(-1,1))). If you do the same for classifiers A and B, you can compare them with same_cell_points = (cm_indices_A == cm_indices_B).all(axis=1) and get the data with X[same_cell_points]. $\endgroup$
    – Matthew
    Jan 7 at 21:32
  • $\begingroup$ That's exactly what I'm looking for! Thanks for your insights, Matthew! $\endgroup$
    – Saul Aryeh Kohn
    Jan 7 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.