0
$\begingroup$

I have a question regarding annotating text data for classification.

Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time to go through all the text samples, but at least a significant portion of them.

  1. Should we focus on generating new samples for each annotator? (They never see the same text samples as any other annotator) (quantity approach).
  2. Or should all annotators see the same samples and the annotator agreement is taken to account? (quality approach).

Thoughts,

  1. will generate more unique samples than 2. (more training samples for a classifier) - and hoping that in the feature extraction part, the useful features will appear by themselves.
  2. will generate less unique samples, but with the annotator agreement taken into account. (less training samples for a classifier, but with higher quality)
Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
2
$\begingroup$

Both metods are against the nature of machine/statistical learning. Lack of data will not let you generalize well in any case. It will have different effect though. Generally speaking:

  1. You will be able to classify more test samples, but sometimes will do it wrong due to errors in data.
  2. You will have precise classification of some test samples, but will have no idea how to classify the rest of them.

So, it's up to your's classification score/quality estimation function which case is better for you.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Well, it's better to annotate some data than nothing at all right ;)? This will be extended with active learning, so it's not against machine learning at all. $\endgroup$
    – Isbister
    Oct 31, 2017 at 12:57
  • $\begingroup$ My point is both ways almost the same. Profit here and loss there or vice versa :) $\endgroup$
    – CrazyElf
    Oct 31, 2017 at 13:04

Not the answer you're looking for? Browse other questions tagged or ask your own question.