I have several annotators who annotated strings of text for me, in order to train an NER model. The annotation is done in json format, and it consists of a string followed by the start and end index of named entities, along with their respective entity type. What is the best way to calculate the IAA score in this case? Is there a tool, or Python library available?
Cohen's kappa is the standard annotation reliability measure for many classification annotation tasks, but it is not a relevant measure for token-level annotation tasks like named entity recognition.
The pairwise F1 measure that disregards out-of-the-span tokens (unannotated tokens) is an ideal measure of annotation reliability for the token-level annotation tasks.
Brandsen A, Verberne S, Lambers K, Wansleeben M, Calzolari N, Béchet F, Blache P, Choukri K, Cieri C, Declerck T, Goggi S. Creating a dataset for named entity recognition in the archaeology domain. InConference Proceedings LREC 2020 2020 (pp. 4573-4577). The European Language Resources Association.
Deleger L, Li Q, Lingren T, Kaiser M, Molnar K, Stoutenborough L, Kouril M, Marsolo K, Solti I. Building gold standard corpora for medical natural language processing tasks. InAMIA Annual Symposium Proceedings 2012 (Vol. 2012, p. 144). American Medical Informatics Association.