For a multi label, multi class categorization on a social media dataset, we have collected around 5000 samples from the dataset and have manually annotated them. 5000 samples are labelled by 3 people, and 1500 of them are same posts. How should I decide the category label now?

For example,

AnnotatorId  Post  Labels
Annotator1: post1: A,B,C
Annotator2: post1: A,D,E
Annotator3: post1: B,D,E

Will the label for post1 be A,B,D through majority voting? Or, is there some better, commonly used approaches?

  • $\begingroup$ Do you know they "quality" of each annotator? You can weight their decisions. It might be good idea to simply merge the labels but that can add too much of noise. Another way would be to use the labels as features for your final classifier; the classifier could deal with noise in features, not in the labels. $\endgroup$ Commented Oct 20, 2015 at 23:28

1 Answer 1


That's one valid way of approaching the problem. In your final solution, though, it will be helpful to quantify the overall inter-rater agreement. For example, Cohen's kappa is a commonly-used metric: \begin{eqnarray} \kappa &=& \frac{p_{o}-p_{e}}{1-p_{e}}\\ &=& 1 - \frac{1-p_{o}}{1-p_{e}}, \end{eqnarray} where $p_{o}$ and $p_{e}$ are the amount of agreement you observe and that due to chance, respectively. The reason this is important is that the amount of agreement your human annotators achieve is a theoretical upper-bound on the performance of your machine learning solution—it provides context for interpreting the performance of your algorithmic approach.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.