Which freely available datasets can I use to train a text classifier?

We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified our content based on a predefined bag of words we can recommend to him engaging content by getting his feedback on random number of posts already classified before.

We can use this info to recommend for him pulses labeled with those classes. But we found If we used a predefined bag of words not related to our content the feature vector will be full of zeros, also categories may be not relevant to our content. so for those reasons we tried another solution that will be clustering our content not classifying it.

Thanks :)

  • 1
    $\begingroup$ I think more details about your problem are necessary before anyone could recommend a dataset. $\endgroup$ – Neil Slater Jun 18 '14 at 16:40
  • 3
    $\begingroup$ For what purpose? Spam filtering? Sentiment analysis? Without a clear purpose it is very difficult to suggest a dataset. $\endgroup$ – lsdr Jun 18 '14 at 18:37
  • $\begingroup$ @lsdr Looking at the answers, it seems that the question does not necessarily need more details. $\endgroup$ – Amir Ali Akbari Jun 19 '14 at 17:25
  • $\begingroup$ @AmirAliAkbari I think they came after an edit. I retracted my close-vote, anyway. $\endgroup$ – Rubens Jun 19 '14 at 17:30
  • $\begingroup$ A more appropriate place for this question is opendata.stackexchange.com $\endgroup$ – sheldonkreger May 29 '15 at 17:38

Some standard datasets for text classification are the 20-News group, Reuters (with 8 and 52 classes) and WebKb. You can find all of them here.

  • $\begingroup$ Thanks :), I already visited it before but I found it's classifications are weak not abstract enough or it may be not related to my content $\endgroup$ – Abdelmawla Jun 21 '14 at 10:25

One of the most widely used test collection for text categorization research (link below). I've used many times. Enjoy your exploration :)

http://www.daviddlewis.com/resources/testcollections/reuters21578/ or http://archive.ics.uci.edu/ml/datasets/Reuters-21578+Text+Categorization+Collection


There is a bunch of datasets made free by UC Irvine to play with here. Among those datasets, there are a few dozen textual datasets that might help you guys with your task.

Those are kind of generic datasets, so depending on your purpose they should not be used as the only data to train your models, or else your model -- while it might work -- will not produce quality results.


Apart from the suggestions above, there is an extremely useful pdf - Benchmarking Text Collections for Classification and Clustering Tasks which contains various datasets along with the benchmarks in order to test our models. This includes 20ng Collection, Reuters and many of the above suggested datasets. I hope it helps!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.