How to improve Vector Space Models with semantic similarity?

Question

I try to construct a classic querying system where I find the most probable candidate text for a query by computing cosine similarities of TFIDF vectors of normalized text of possible answers. This works quite well if the query and the candidate texts contain identical (normalized) words. So, the following question:

When was the King of England assassinated?

my system correctly finds the following closest answer from the corpus:

The King of England was assassinated in 1234.

So far, so good. My problem is that I want to find this answer in the case of questions with the same meaning but slightly different synonym wording, like:

When was the British Monarch killed?

How could I compute similarity taking semantic distance into account? Shall I use word2vec representation instead of TFIDF?

Answer 1

You are right. TF-IDF is not suitable for your problem statement because TF-IDF tends to assign lower weight to common words appearing in two documents while giving importance to important words. So while

When was the King of England assassinated?

and

The King of England was assassinated in 1234.

will match because of words like King and England but

When was the British Monarch killed?

wont match as the words British and Monarch are one of its kind in the corpus.

A better approach is doc2vec technique which converts documents into vectors and predicts cosine similarity. It works very well in a very large number of documents or corpus, containing tens of thousands of documents. Doc2Vec is a generalization of Word2Vec which can give very interesting relationship between words. For example, king - man + woman = queen without being implicitly or explicitly told.