I have a massive text block of poorly written statements, some facts, some opinions. It's all scraped from various public internet resources and all in english.

I am now using nltk in python to try to work out some basic statements from this massive text block.

My goal is not to capture all statements, false positives are not too bad either. I would simply like to get a few probably true and simple assertions from this text block.


Bob is married to Lisa since 1983.

Lisa's cat Snowball died two years ago.

Bob and Lisa has a 30 year old son named George.

From these simple sentences, I would like to parse out some simple facts, like:

"Bob is married", "Lisa is married", "George is Lisa's son", "George is Bobs son", "Snowball is a cat"

It does not need to be exactly this.

Actual question

What is this kind of processing called? (Except for that it is a subcategory of Natural Language Processing.)

What are the simpler and the harder approaches to this problem?

How well has this problem been solved by public and proprietary algorithms?

  • $\begingroup$ @OvisAmmon I so love sarcastic begging for points that now I feel obligated to wait just a little bit longer to see if I can get an even better answer, which is what I was doing anyway. $\endgroup$
    – firelynx
    Nov 4 '16 at 14:44

You may want to read about the task of Relation Extraction. The most common scenario is to extract triplets (entity1, relation, entity2). Usually entity1 and entity2 are given and the list of possible relations is known. Common methods use a classifier, others can be rule-based.

With your example from the sentence "Bob is married to Lisa since 1983." the system can extract (Bob, is_married, Lisa)

A variant is Open Information Extraction which tries to extract relations but with no information specified.

There exist multiple projects on the subject such as DeepDive, IEPY or openie-standalone.

This this presentation or this review give an overview of the task.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.