18
$\begingroup$

I'm currently searching for labeled datasets to use to train a model to extract named entities from informal text (think something similar to tweets). Because capitalization and grammar are often lacking in the documents in my dataset, I'm looking for out of domain data that's a bit more "informal" than the news articles and journal entries that many of today's state of the art named entity recognition systems are trained on. Any recommendations? So far I've only been able to locate 50k tokens from twitter published here: https://github.com/aritter/twitter_nlp/blob/master/data/annotated/ner.txt

$\endgroup$
6
$\begingroup$

As I understand it, these are the properties that you're seeking in a sample dataset:

  1. Text data
  2. It should be informal, i.e. have typos, slang, and basically something not professionally edited
  3. Something other than Twitter (I don't blame you, Twitter is a useful yet way overused example datasource in text mining)

Here are some recommendations:

  1. Emails from the SpamAssassin corpus -- note that both "ham" (non-spam) and spam datasets are available
  2. microblogPCU data set from UCI, which is data scraped from the microblogs of Sina Weibo users -- note, the raw text data is a mix of Chinese and English (you could perform machine translation of the Chinese, filter to only English, or use it as-is)
  3. Amazon Commerce reviews dataset from UCI
  4. Within the bag-o-words dataset, try using the Enron emails
  5. The Twenty Newsgroups dataset
  6. This nice collection of SMS spam
  7. You can always scrape (extract) your own text data from the Internet; I'm not sure which language or statistical package you're using, but XPath-based packages are available in R (rvest, scrapeR, etc) and Python to accomplish this
$\endgroup$
  • 1
    $\begingroup$ Are any of these datasets annotated with named entities though? I believe that's what OP was looking for. $\endgroup$ – Mr. Phil Feb 28 '17 at 19:36
3
$\begingroup$

Check these :

Repository of Test Domains for Information Extraction : http://www.isi.edu/info-agents/RISE/repository.html

DBpedia : http://wiki.dbpedia.org/Downloads32 (mirror)

Link Updated :

http://www.isi.edu/integration/RISE/

https://github.com/dbpedia/extraction-framework/wiki/The-DBpedia-Data-Set

$\endgroup$
  • 1
    $\begingroup$ Please update these links as none of them is working anymore. $\endgroup$ – Mr. Phil Feb 28 '17 at 19:31
0
$\begingroup$

Some of the sources that I have used:

I think these datasets will be of great help for your task

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.