1
$\begingroup$

I'm extracting company names from text blobs.

I'm trying to calculate recall.

Confusion matrices makes more sense to me from a binary classification point of view, so I was wondering if the following "match categories" made sense. E.g. the second last row could both be a false positive and a false negative?

(Before I had a go at this, I thought extraction would basically boil down to TP and FP: either the human and machine agrees, or they don't.)

enter image description here

Here Google talks about how rows can be FP and FN at the same time: https://cloud.google.com/document-ai/docs/workbench/evaluate#individual-labels

Note: False positives and false negatives are not mutually exclusive. For example, if a predicted entity has a corresponding, albeit incorrect, annotation in the test set, then there will be one false positive for the predicted entity and one false negative for the annotation. A predicted entity without an annotation is only a false positive. An annotation without an associated prediction is only a false negative.

$\endgroup$
1
  • $\begingroup$ After rereading, I realized what's the mix up. A sample pair results in one occurrence in the confusion matrix, however what is a false positive to one class (column), may be a false negative to another class. I updated my answer to better address this and your specific example. Google's interpretation on how to handle missing predictions or annotations is more of an opinion that anything missing is a Negative never positive, as this would differ from the case of treating missing values as their own class. $\endgroup$
    – prijatelj
    Commented Oct 26, 2023 at 16:26

2 Answers 2

1
$\begingroup$

In general, confusion matrices are for classification tasks, to see the data where a model makes correct and incorrect predictions, but not only for binary tasks. For example, here is a confusion matrix on MNIST classification taken from the book by A. Geron: enter image description here

When you talk about false positives and false negatives overlapping, as you point out it means that your data is incorrectly labeled. If you are extracting names of companies, what are your predictions compared against?

$\endgroup$
3
  • $\begingroup$ "but not only for binary tasks" -> this is at least ambiguous, confusion matrices are commonly used with binary tasks. $\endgroup$
    – Erwan
    Commented Oct 25, 2023 at 17:11
  • $\begingroup$ I don't see your point. They are certainly used for binary tasks, but not only for binary tasks. Where is your confusion? (pardon the pun) $\endgroup$ Commented Oct 26, 2023 at 7:30
  • $\begingroup$ sorry, I don't know how but I misread your sentence yesterday. My mistake. $\endgroup$
    – Erwan
    Commented Oct 26, 2023 at 16:36
1
$\begingroup$

In Brief: Can a sample be both FN and FP? No in Binary, Yes in Multiclass

This is a problem of interpretation of what matters or what is the focus as in "these classes are Positive and rest Negative".

No, not within the same binary confusion matrix.

A single sample pair (human, machine) results in a single occurrence added to the confusion matrix, like adding a tally to a board.

In binary classification, your confusion matrix will only consist of the classic 4 possibilities forming a 2x2 matrix:

Where the Human serves as the accepted source of "truth"

Total Population Machine: Positive Machine: Negative
Human: Positive True Positive False Negative
Human: Negative False Positive True Negative

This is a recreation of a table from Confusion Matrix page on Wikipedia

To view those labels in the example you provided as a binary classification problem, you must reduce the labels into a binary outcome. Once you have an appropriate mapping of binary state, you get the above matrix, and a single occurrence cannot be both FN and FP.

Yes, when measuring recall over multiple classes

To estimate recall $= \frac{TP}{P} = \frac{TP}{TP + FN}$,, thus rows in our confusion matrix with humans as truth. Straight forward for the binary case.

To estimate recall for a multiclass classification, you would calculate the recall per row of cm, then take their average. Recall = True Positive Rate. This is how scikit-learn.metrics.recall_score measures it, and has a parameter for how to conduct the average.

Your example has the following CM with Humans as ground truth (rows) and the companies indicated by the first letter of their name and the missing value as $\emptyset$.

T W M A $\emptyset$
T 0 0 0 0 2
W 0 1 0 0 0
M 0 0 1 0 0
A 1 0 0 0 0
$\emptyset$ 1 0 0 0 0

Measuring the recall per class (row) would mean to consider for one row a false negative what would be considered a false positive for another class.


Given it seems there is not a current binary mapping of those company names, if you view this as a binary evaluation as you initially did, you may only check if the machine matches the human (if the human is deemed as the "truth"), and so you only get True Positive for a match, and as there are only Positives (single row, 1x2 matrix), FP = 0 and FN >= 0. Incorrect matches all in FN, because the source of truth never has a "negative" value, it only specifies what is positive. In this case recall is equivalent to accuracy, though is not really a confusion matrix.


In general, I typically recommend Matthews Correlation Coefficient, which is a multiclass classification generalization of Pearson's Correlation Coefficient, as it is a measure that uses the whole confusion matrix, rather than only parts. Scikit learn has this too. Though use whatever measure fits your purpose.

More on Multiclass Classification Evaluation with a Confusion Matrix

As noted in brewmaster321's answer, you can use a confusion matrix for multiple class classification, which better suits your example. This way, an occurrence will go to the: Machine said Target, Human said Apple Inc. box, instead of some binary class that loses that possibly useful information.

Out of convenience on my part, here is a screenshot of an informative visual I made for a paper of mine currently under review. We dealt with both binary and multiclass classification together. Binary case of Known vs Unknown, but where there were multiple classes within their corresponding known and unknown set. Visual depiction of a multi-class classification confusion matrix being collapsed into a binary class confusion matrix

The most informative confusion matrix is thus the "raw" multi-class confusion matrix, where there are multiple classes in both known & unknown sets. When you focus on only binary labelings, as done in the "Novelty Detection Confusion Matrix", you lose the information about confusion of the machine vs the human within those binary labels.

Given your example has blanks for the machine, while labels are present for the human, you will have to decide how those missing values are handled. I recommend a "missing" label ($\emptyset$) instead of "unknown" as used above, as this "unknown" refers to an "other class mutually exclusive to known classes", rather than a missing or unavailable prediction.

Further Reading

References

  1. Arxiv preprint of my paper, Figure 4, https://arxiv.org/abs/2212.12141 under CC-BY 4.0 : https://creativecommons.org/licenses/by/4.0/
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.