Choosing class labels from annotated data

Question

For a multi label, multi class categorization on a social media dataset, we have collected around 5000 samples from the dataset and have manually annotated them. 5000 samples are labelled by 3 people, and 1500 of them are same posts. How should I decide the category label now?

For example,

AnnotatorId  Post  Labels
-------------------------
Annotator1: post1: A,B,C
Annotator2: post1: A,D,E
Annotator3: post1: B,D,E

Will the label for post1 be A,B,D through majority voting? Or, is there some better, commonly used approaches?

Answer 1

That's one valid way of approaching the problem. In your final solution, though, it will be helpful to quantify the overall inter-rater agreement. For example, Cohen's kappa is a commonly-used metric: \begin{eqnarray} \kappa &=& \frac{p_{o}-p_{e}}{1-p_{e}}\\ &=& 1 - \frac{1-p_{o}}{1-p_{e}}, \end{eqnarray} where $p_{o}$ and $p_{e}$ are the amount of agreement you observe and that due to chance, respectively. The reason this is important is that the amount of agreement your human annotators achieve is a theoretical upper-bound on the performance of your machine learning solution—it provides context for interpreting the performance of your algorithmic approach.