For a multi label, multi class categorization on a social media dataset, we have collected around 5000 samples from the dataset and have manually annotated them. 5000 samples are labelled by 3 people, and 1500 of them are same posts. How should I decide the category label now?
For example,
AnnotatorId Post Labels
-------------------------
Annotator1: post1: A,B,C
Annotator2: post1: A,D,E
Annotator3: post1: B,D,E
Will the label for post1 be A,B,D
through majority voting? Or, is there some better, commonly used approaches?