I have several annotators who annotated strings of text for me, in order to train an NER model. The annotation is done in json format, and it consists of a string followed by the start and end index of named entities, along with their respective entity type. What is the best way to calculate the IAA score in this case? Is there a tool, or Python library available?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Cohen's kappa is the standard annotation reliability measure for many classification annotation tasks, but it is not a relevant measure for token-level annotation tasks like named entity recognition.
The pairwise F1 measure that disregards out-of-the-span tokens (unannotated tokens) is an ideal measure of annotation reliability for the token-level annotation tasks.
Bibliography
Brandsen A, Verberne S, Lambers K, Wansleeben M, Calzolari N, Béchet F, Blache P, Choukri K, Cieri C, Declerck T, Goggi S. Creating a dataset for named entity recognition in the archaeology domain. InConference Proceedings LREC 2020 2020 (pp. 4573-4577). The European Language Resources Association.
Deleger L, Li Q, Lingren T, Kaiser M, Molnar K, Stoutenborough L, Kouril M, Marsolo K, Solti I. Building gold standard corpora for medical natural language processing tasks. InAMIA Annual Symposium Proceedings 2012 (Vol. 2012, p. 144). American Medical Informatics Association.
$\begingroup$
$\endgroup$
I think the Kappa coefficient is the most commonly used to measure inter-annotator agreement, but there are other options as well.
sklearn provides an implementation of the Cohen Kappa coefficient, which can be used to compare two annotators.
