13
$\begingroup$

I'm creating a simple chatbot. I want to obtain the information from the user response. An example scenario:

Bot : Hi, what is your name?
User: My name is Edwin.

I wish to extract the name Edwin from the sentence. However, the user can response in different ways such as 

User: Edwin is my name.
User: I am Edwin.
User: Edwin.

I'm tried to rely on the dependency relations between words but the result does not do well.

Any idea on what technique I could use to tackle this problem?

[UPDATED]

I tested with named entity recognition together with part of speech tagger and parser. I found out that most model is trained in a way that the first character of the entity for the person name or the proper noun must be upper case. This may be true for normal document, but it is irrelevant for a chatbot. E.g. 

User: my name is edwin.

Most NER failed to recognize this.

Improve this question
$\endgroup$
2
  • $\begingroup$ This explains how modern chat bots are built, but I wouldn't call it simple. You can learn more by searching for "question answering". $\endgroup$
    – Emre
    Commented Nov 2, 2016 at 8:36
  • $\begingroup$ I so like how people asking for questions and receiving answers mark an answer as acceptable one :P $\endgroup$
    – chewpakabra
    Commented Nov 4, 2016 at 11:50

4 Answers 4

Reset to default
7
$\begingroup$

You can possibly use a combination of Named Entity Recognition and Syntactical Analysis - while the word Edwin is certainly propping up, imagine a situation where the name is Edward Philip Martel. NER detects each word as a separate entities (hence 3 different entities) - thus, you will anyways have to string them together based on some logic. Further, in the case of multiple names being present, it can get harder to disambiguate (e.g. John & Ramsey dined at Winterfell).

This is where the analysis of the sentence syntax would also help (assuming that the end user enters a relatively coherent and proper sentence - if slang and short forms of text are used, even the Stanford NLP can help upto a certain extent only).

One way of leveraging on syntax analysis / parsing and NER is in the following examples -

 1. User: Edwin is my name.
 2. User: I am Edwin.
 3. User: My name is Edwin.

In each of the cases (as is generically the case as well), the Entity Name (Proper Noun / Noun) is associated in close proximity to a Verb. Hence, if you first parse the sentence to determine verbs and then apply NER to surrounding (+/- 1 or 2) words, you may have a relatively decent way to resolve the problem. This solution would depend primarily on the syntax rules you create to identify NERs as well as the window around the verbs.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ You possibly will implement a 'non deterministic finite automata ', where each sentence is a response that a pattern accepts. Some grammars are implemented on something like this. (NLP/Grammar). If you need how to do this, look at the framework stanfordnlp.github.io/CoreNLP $\endgroup$
    – Intruso
    Commented Nov 7, 2016 at 12:16
5
$\begingroup$

You should use Named Entity Recognition, for example from NLTK. You can find a usage example here. It would work for your described case quite well.

Improve this answer
$\endgroup$
4
$\begingroup$

This can be easily done with CRFs. You can use BIO encoding to tag your sentence. Then pass it to CRFs. You just have to create a few tagged sentences for training purpose as follows,

 I am Edwin.
 O O  B-NAME

 You can call me Alfred
 O   O    O    O B-NAME

 My name is  Edwin   thomas
 O  O     O  B-NAME  I-NAME

CRFsuite and CRF++ are some of the good implementations. CRFsuite has a python wrapper called pycrfsuite, which is quite easy to implement. Check this ipython notebook or this code snippet on github for end-to-end implementation of NER.

check this Open source Chat bot project on github with NER and Intent Classification written in python. They have an easy to use training UI where you can train your bot to extract information from sentences.

Improve this answer
$\endgroup$
0
$\begingroup$

In order to perform such tasks with high accuracy, I suggest you to build a LSTM model with word embeddings with the help of word2vec. LSTMs can help to retrieve information from sentence as well as predict the next character or word given a set of words is already present in the sentence.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.