Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
I have a question regarding annotating text data for classification.
Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time to go through all the text samples, but at least a significant portion of them.
Should we focus on generating new samples for each annotator? (They never see the same text samples as any other annotator) (quantity approach).
Or should all annotators see the same samples and the annotator agreement is taken to account? (quality approach).
will generate more unique samples than 2. (more training samples for a classifier) - and hoping that in the feature extraction part, the useful features will appear by themselves.
will generate less unique samples, but with the annotator agreement taken into account. (less training samples for a classifier, but with higher quality)