21
$\begingroup$

I'm currently searching for labeled datasets to train a model to extract named entities from informal text (something similar to tweets). Because capitalization and grammar are often lacking in the documents in my dataset, I'm looking for out of domain data that's a bit more "informal" than the news article and journal entries that many of today's state of the art named entity recognition systems are trained on.

Any recommendations? So far I've only been able to locate 50k tokens from twitter published here.

Improve this question
$\endgroup$
6

3 Answers 3

Reset to default
6
$\begingroup$

As I understand it, these are the properties that you're seeking in a sample dataset:

  1. Text data
  2. It should be informal, i.e. have typos, slang, and basically something not professionally edited
  3. Something other than Twitter (I don't blame you, Twitter is a useful yet way overused example datasource in text mining)

Here are some recommendations:

  1. Emails from the SpamAssassin corpus -- note that both "ham" (non-spam) and spam datasets are available
  2. microblogPCU data set from UCI, which is data scraped from the microblogs of Sina Weibo users -- note, the raw text data is a mix of Chinese and English (you could perform machine translation of the Chinese, filter to only English, or use it as-is)
  3. Amazon Commerce reviews dataset from UCI
  4. Within the bag-o-words dataset, try using the Enron emails
  5. The Twenty Newsgroups dataset
  6. This nice collection of SMS spam
  7. You can always scrape (extract) your own text data from the Internet; I'm not sure which language or statistical package you're using, but XPath-based packages are available in R (rvest, scrapeR, etc) and Python to accomplish this
Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Are any of these datasets annotated with named entities though? I believe that's what OP was looking for. $\endgroup$
    – Mr. Phil
    Feb 28, 2017 at 19:36
3
$\begingroup$

Check these :

Repository of Test Domains for Information Extraction : http://www.isi.edu/info-agents/RISE/repository.html

DBpedia : http://wiki.dbpedia.org/Downloads32 (mirror)

Link Updated :

http://www.isi.edu/integration/RISE/

https://github.com/dbpedia/extraction-framework/wiki/The-DBpedia-Data-Set

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Please update these links as none of them is working anymore. $\endgroup$
    – Mr. Phil
    Feb 28, 2017 at 19:31
0
$\begingroup$

Some of the sources that I have used:

I think these datasets will be of great help for your task

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.