1
$\begingroup$

I'm trying to compare two NER tools on an annotated corpus and I'm not sure which is the best metric to use, as I haven't worked with NER models before. To be more specific, I'm interested in one class only, so I want to evaluate them on that particular class.

$\endgroup$
1
$\begingroup$

A good starting point is to look at the evaluation measures used in the NER shared tasks: https://nlpprogress.com/english/named_entity_recognition.html.

Generally the F1-score can be used for one specific class, but there are different options regarding what is counted as an instance:

  • every occurrence of the full NE. In this case any difference between the predicted and the gold is counted as false, even if it's only one token difference.
  • every token in an entity. In this case a partially matched entity counts as "partially correct": if a word is predicted outside instead of inside, it's a false negative and conversely.
  • Other variants:
    • count only unique entities, in order to observe the diversity of the entities recognized.
    • count only entities which didn't appear in the training set, to observe the generalization power.

(writing this from memory, I could miss something)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.