# Choosing class labels from annotated data

For a multi label, multi class categorization on a social media dataset, we have collected around 5000 samples from the dataset and have manually annotated them. 5000 samples are labelled by 3 people, and 1500 of them are same posts. How should I decide the category label now?

For example,

AnnotatorId  Post  Labels
-------------------------
Annotator1: post1: A,B,C
Annotator2: post1: A,D,E
Annotator3: post1: B,D,E


Will the label for post1 be A,B,D through majority voting? Or, is there some better, commonly used approaches?

• Do you know they "quality" of each annotator? You can weight their decisions. It might be good idea to simply merge the labels but that can add too much of noise. Another way would be to use the labels as features for your final classifier; the classifier could deal with noise in features, not in the labels. – Vladislavs Dovgalecs Oct 20 '15 at 23:28

That's one valid way of approaching the problem. In your final solution, though, it will be helpful to quantify the overall inter-rater agreement. For example, Cohen's kappa is a commonly-used metric: \begin{eqnarray} \kappa &=& \frac{p_{o}-p_{e}}{1-p_{e}}\\ &=& 1 - \frac{1-p_{o}}{1-p_{e}}, \end{eqnarray} where $p_{o}$ and $p_{e}$ are the amount of agreement you observe and that due to chance, respectively. The reason this is important is that the amount of agreement your human annotators achieve is a theoretical upper-bound on the performance of your machine learning solution—it provides context for interpreting the performance of your algorithmic approach.