I'm trying to compare two NER tools on an annotated corpus and I'm not sure which is the best metric to use, as I haven't worked with NER models before. To be more specific, I'm interested in one class only, so I want to evaluate them on that particular class.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
A good starting point is to look at the evaluation measures used in the NER shared tasks: https://nlpprogress.com/english/named_entity_recognition.html.
Generally the F1-score can be used for one specific class, but there are different options regarding what is counted as an instance:
- every occurrence of the full NE. In this case any difference between the predicted and the gold is counted as false, even if it's only one token difference.
- every token in an entity. In this case a partially matched entity counts as "partially correct": if a word is predicted outside instead of inside, it's a false negative and conversely.
- Other variants:
- count only unique entities, in order to observe the diversity of the entities recognized.
- count only entities which didn't appear in the training set, to observe the generalization power.
(writing this from memory, I could miss something)
