I have a question regarding annotating text data for classification.
Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time to go through all the text samples, but at least a significant portion of them.
- Should we focus on generating new samples for each annotator? (They never see the same text samples as any other annotator) (quantity approach).
- Or should all annotators see the same samples and the annotator agreement is taken to account? (quality approach).
- will generate more unique samples than 2. (more training samples for a classifier) - and hoping that in the feature extraction part, the useful features will appear by themselves.
- will generate less unique samples, but with the annotator agreement taken into account. (less training samples for a classifier, but with higher quality)