2
$\begingroup$

I have several annotators who annotated strings of text for me, in order to train an NER model. The annotation is done in json format, and it consists of a string followed by the start and end index of named entities, along with their respective entity type. What is the best way to calculate the IAA score in this case? Is there a tool, or Python library available?

$\endgroup$
0
$\begingroup$

I think the Kappa coefficient is the most commonly used to measure inter-annotator agreement, but there are other options as well.

sklearn provides an implementation of the Cohen Kappa coefficient, which can be used to compare two annotators.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.