-1
$\begingroup$

I try to construct a classic querying system where I find the most probable candidate text for a query by computing cosine similarities of TFIDF vectors of normalized text of possible answers. This works quite well if the query and the candidate texts contain identical (normalized) words. So, the following question:

When was the King of England assassinated?

my system correctly finds the following closest answer from the corpus:

The King of England was assassinated in 1234.

So far, so good. My problem is that I want to find this answer in the case of questions with the same meaning but slightly different synonym wording, like:

When was the British Monarch killed?

How could I compute similarity taking semantic distance into account? Shall I use word2vec representation instead of TFIDF?

$\endgroup$
  • $\begingroup$ A document embedding will be an improvement, but read the literature; question answering has seen a lot of research recently. $\endgroup$ – Emre Jun 22 '17 at 7:54
  • $\begingroup$ if you have sufficient question-answer pair then you can look at Dynamic memory networks , Recently facebook released a paper for question - answering system . $\endgroup$ – Abhishek Verma Jul 23 '17 at 10:08
0
$\begingroup$

You are right. TF-IDF is not suitable for your problem statement because TF-IDF tends to assign lower weight to common words appearing in two documents while giving importance to important words. So while

When was the King of England assassinated?

and

The King of England was assassinated in 1234.

will match because of words like King and England but

When was the British Monarch killed?

wont match as the words British and Monarch are one of its kind in the corpus.

A better approach is doc2vec technique which converts documents into vectors and predicts cosine similarity. It works very well in a very large number of documents or corpus, containing tens of thousands of documents. Doc2Vec is a generalization of Word2Vec which can give very interesting relationship between words. For example, king - man + woman = queen without being implicitly or explicitly told.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.