I have created a domain-specific dataset, lets say it is relating to python programming topic posts. I have taken data from various places specific to this topic to create positive examples in my dataset. For example, python related subreddits, stack exchange posts tagged with python, twitter posts hashtagged with python or python specific sites.

The data points taken from these places are considered positive data points and then I have retrieved data points from the same sources but relating to general topics, searched if they contain the word python in them and if they do discard them to create the negative examples in my dataset.

I have been told that I can use the training set from the dataset as is, but that I need to manually annotate the test set for the results to be valid, otherwise they would be biased. Is this correct? How would they be biased? To be clear the test set contains different entries to the training set.

There are close to 200,000 entries in the test set which makes manual annotation difficult. I have seen similar methods been used in papers I have previously read without mention of manual annotation. Is this technique valid or do I have to take some extra steps to ensure the validity of the test sets?


There are two potential biases:

  • With this automatic method, you might have a few erroneous labels. For example it happens regularly here on DataScienceSE that a user tags a question "python" but actually the question is not specific to python at all. Same thing for the opposite case: for example it's possible that some content contains some python code but doesn't mention "python" anywhere.
  • The distribution between positive and negative classes is arbitrary. Let's assume you use 50% positive / 50% negative: if later you want to apply your classifier on a new data science website where only 10% of the content is about python, it's likely to predict a lot of false positive cases so the true performance on this data will be much lower than on your test set.

It's rare to have a perfect dataset, so realistically in my opinion the first issue is probably acceptable because the noise in the labels should be very limited. The second issue could be a bit more serious but this depends on what is the end application. Keep in mind that a trained model is meant to be applied to the same kind of data as it was trained/tested on.

| improve this answer | |
  • $\begingroup$ Thank you for your answer. For the issue of noise presented in the first example is there an option other than manual annotation to make sure of the integrity of the test set. I have a reviewer which says the results are biased and cannot be used. $\endgroup$ – dmnte Aug 1 at 23:38
  • $\begingroup$ @dmnte obviously you're not going to annotate manually 200k instances, however you could measure the amount of noise in the labels on a sample: you take for instance 500 random instances (it's important to pick them randomly), annotate these manually and count how many were wrongly labelled with the automatic method. It's imperfect but if you find that there is only a small number of errors, it's evidence that your automatic labeling is reliable. If you find a large number of errors, then the reviewer is right unfortunately! $\endgroup$ – Erwan Aug 2 at 1:16
  • $\begingroup$ thanks, your answers have been very helpful. $\endgroup$ – dmnte Aug 2 at 6:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.