Text annotating process, quality vs quantity?

Question

I have a question regarding annotating text data for classification.

Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time to go through all the text samples, but at least a significant portion of them.

1. Should we focus on generating new samples for each annotator? (They never see the same text samples as any other annotator) (quantity approach).
2. Or should all annotators see the same samples and the annotator agreement is taken to account? (quality approach).

Thoughts,

1. will generate more unique samples than 2. (more training samples for a classifier) - and hoping that in the feature extraction part, the useful features will appear by themselves.
2. will generate less unique samples, but with the annotator agreement taken into account. (less training samples for a classifier, but with higher quality)

Answer 1

It depends on the context which you consider. For example, suppose there is a situation that all possible states can be covered by 10K different texts. It is trivial, if these texts are all annotated, then for 1000 tests, at least we can classify 500 of them truly (as we have two classes, and probability of wrong annotation for each text is at most 0.5).

Now, suppose 1K of 10K texts are annotated. Then, as annotations are exact, we can classify 1/10 of 1000 texts truly (because we have no idea about the other 9K possible states).

Therefore, in this situation quantity is more important than quality.

Also, we can consider these cases, when the possible states are 1K. It could be straightforward to show that in this case (if the power of annotators is same as the former case) quality can be more important than quantity. However, in the most cases this number is not realistic.

In sum, as in the most cases the variety of texts are more than the power of annotators, we prefer quantity to quality, as we can cover more text space and machine can learn more. Although the accuracy can be less, but for two classes classification is negligible.