0
$\begingroup$

I am working on accumulating a large database of labeled sentences for several projects/experiments. At present I am only using Wikipedia and Project Gutenberg as sources of data. Between these two sources I expect I will be able to extract millions of sentences. Labeling them may be another matter, but I'm less concerned about that right now.

My chief need is to find a broader variety of sentences. Questions, exclamations, invective, conversation, and so on. My project is several deep learning experiments to look at different types of sentences and structures. It would be helpful if the sentences were labeled.

The ultimate goal of my project is to be able to infer the type, structure, and coherence of a sentence (with separate models). Generating gibberish is pretty easy to do, so I'm not as concerned about finding sources of non-sense sentences.

My current concern is that the vast number of sentences are declarative (statements) rather than interrogatives, exclamatory or imperatives. This bias towards declarative sentences means that my sample sizes will be limited unless I can find sources of questions, exclamations and commands.

Alternatively, I would be open to suggestions about how to generate the data I need.

$\endgroup$
1
  • 2
    $\begingroup$ Interview transcripts? Try a news source. Welcome to the site and good luck! $\endgroup$
    – Emre
    Commented Aug 14, 2018 at 18:50

1 Answer 1

0
$\begingroup$

I ultimately settled on Gutenberg Project as a source of massive amounts of data. I realized that books (fiction and non-fiction) cover pretty much every permutation of human communication that can be represented in plain text. I've written a couple of scripts to parse Gutenberg text files and next I'll be working on a support vector machine to classify each passage as regular content or other (such as headers, titles, etc).

https://www.gutenberg.org/wiki/Gutenberg:Information_About_Robot_Access_to_our_Pages

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.