Word2Vec - document similarity

Question

Lets say I have text data for different documents from 2005 - 2015. I want to compare the similarity between $t$ and $t-1$ documents. So I take the document at 2006 and compare it with the document at 2005, take the document at 2007 and compare with the document at 2006 … all the way to 2015, compared with 2014.

I have computed for each year a Word2Vec model independently of other year and obtained high dimensional arrays for each Word. So I have 10 Word2Vec models from 2005 - 2015.

Whats the best way for me to compare the documents similarity from here.

Previously I used TF-IDF where I was able to for each document have a large matrix with words in the rows and documents in the columns. Then I could combine he TermDocumentMatrix at $t$ and $t-1$ and compute the cosine similarity.

However Word2Vec gives much high dimensional arrays and I cannot think how to compare W2V models from $t$ to $t-1$.

Any help would be great!

Answer 1

you might want to look into 'approximate nearest neighbors' analysis, and particularly the annoy package from Spotify:

https://github.com/spotify/annoy

It's not as precise as an exact comparison, but it may be close enough to get you what you need.

Answer 2

Although there are several nearest neighbor tool as one mention by @oneextrafact. The problem with that tool is you need to index the external database for a logic operation like you mentioned that you want to build some logic using over date. I will recommend you is to extract a document vector. Although there are several approaches to do that either by training doc2vec or using pre-trained BERT. In case you want to use word2vec then I will recommend the following approach.

1. Train word2vec
2. extract important phrases from each document using text rank.
3. Set a thresh hold for text rank score(0-1) and retrieve those phrases and tokenize those phrases
4. extract word embeddings of those words and average out or sum the vector
5. Now that vector represents the embeddings of that document. Place that vector along with document text, id, and year in Elastic search version 7 and above
6. Elastic search support nearest neighbor search along with other logic like filtering Dates etc.