# Extract information from sentence

I'm creating a simple chatbot. I want to obtain the information from the user response. An example scenario:

Bot : Hi, what is your name?
User: My name is Edwin.


I wish to extract the name Edwin from the sentence. However, the user can response in different ways such as

User: Edwin is my name.
User: I am Edwin.
User: Edwin.


I'm tried to rely on the dependency relations between words but the result does not do well.

Any idea on what technique I could use to tackle this problem?

[UPDATED]

I tested with named entity recognition together with part of speech tagger and parser. I found out that most model is trained in a way that the first character of the entity for the person name or the proper noun must be upper case. This may be true for normal document, but it is irrelevant for a chatbot. E.g.

User: my name is edwin.


Most NER failed to recognize this.

• This explains how modern chat bots are built, but I wouldn't call it simple. You can learn more by searching for "question answering". – Emre Nov 2 '16 at 8:36
• I so like how people asking for questions and receiving answers mark an answer as acceptable one :P – chewpakabra Nov 4 '16 at 11:50

You can possibly use a combination of Named Entity Recognition and Syntactical Analysis - while the word Edwin is certainly propping up, imagine a situation where the name is Edward Philip Martel. NER detects each word as a separate entities (hence 3 different entities) - thus, you will anyways have to string them together based on some logic. Further, in the case of multiple names being present, it can get harder to disambiguate (e.g. John & Ramsey dined at Winterfell).

This is where the analysis of the sentence syntax would also help (assuming that the end user enters a relatively coherent and proper sentence - if slang and short forms of text are used, even the Stanford NLP can help upto a certain extent only).

One way of leveraging on syntax analysis / parsing and NER is in the following examples -

 1. User: Edwin is my name.
2. User: I am Edwin.
3. User: My name is Edwin.


In each of the cases (as is generically the case as well), the Entity Name (Proper Noun / Noun) is associated in close proximity to a Verb. Hence, if you first parse the sentence to determine verbs and then apply NER to surrounding (+/- 1 or 2) words, you may have a relatively decent way to resolve the problem. This solution would depend primarily on the syntax rules you create to identify NERs as well as the window around the verbs.

• You possibly will implement a 'non deterministic finite automata ', where each sentence is a response that a pattern accepts. Some grammars are implemented on something like this. (NLP/Grammar). If you need how to do this, look at the framework stanfordnlp.github.io/CoreNLP – Intruso Nov 7 '16 at 12:16

You should use Named Entity Recognition, for example from NLTK. You can find a usage example here. It would work for your described case quite well.

This can be easily done with CRFs. You can use BIO encoding to tag your sentence. Then pass it to CRFs. You just have to create a few tagged sentences for training purpose as follows,

 I am Edwin.
O O  B-NAME

You can call me Alfred
O   O    O    O B-NAME

My name is  Edwin   thomas
O  O     O  B-NAME  I-NAME


CRFsuite and CRF++ are some of the good implementations. CRFsuite has a python wrapper called pycrfsuite, which is quite easy to implement. Check this ipython notebook or this code snippet on github for end-to-end implementation of NER.

check this Open source Chat bot project on github with NER and Intent Classification written in python. They have an easy to use training UI where you can train your bot to extract information from sentences.

In order to perform such tasks with high accuracy, I suggest you to build a LSTM model with word embeddings with the help of word2vec. LSTMs can help to retrieve information from sentence as well as predict the next character or word given a set of words is already present in the sentence.