Weighted sum of word vectors for document similarity

Question

I have trained a word2vec model on a corpus of documents. I then compute the term frequency (the same Tf in TfIDF) of each word in each document, multiply each words Tf by its corresponding word vector (this is the weighted part), and sum each of these weighted vectors together element wise to obtain a single vector for a document.

Is this method valid?

An example, to make it clearer. Take the document:

"The car drives on the road"

The Tf for each word would be {'the':0.3333, 'car':0.1666, 'drives':0.1666, 'on':0.1666, 'road':0.16666} (obtained by taking the word count of a word and dividing by the total number of words in the document). If we have a trained word2vec model, we can do the following

$$0.333*\begin{bmatrix} the_0 \\ the_1 \\ \vdots \\ the_n \end{bmatrix} + 0.1666*\begin{bmatrix} car_0 \\ car_1 \\ \vdots \\ car_n \end{bmatrix} + ...$$

where each of the column vectors is the word vector for that word. The final result is an $n\times 1$ dimensional vector representing the document.

$$ \begin{bmatrix} 0.333*the_0 + 0.1666*car_0 + \dots \\ 0.333*the_1 + 0.1666*car_1 + \dots \\ \vdots \\ 0.333*the_n + 0.1666*car_n + \dots \end{bmatrix} $$

I appreciate there are other methods such as doc2vec that aim to do very much the same, but in a much more sophisticated way. But is my method valid / is there anything blatantly wrong here?

I have tested this method and even used some document similarity metrics (yes, after normalizing the vectors of course), and yielded some good results for my industrial application. But I want to know more about the academics of this method.

The nice thing about this way, is that by using the word2vec vectors, similarity queries between documents yield very good results due to semantic similarities (euclidean closeness) between word vectors, even if different words are used across documents; this is something TfIDF cannot do as each word is treated differently.

Thanks in advance!

Answer 1

Yes, your method is valid and it has been studied before it is known as Mean of Word Embeddings (MOWE) or Sum of Word Embeddings (SOWE), although your method is more a weighted mean of vectors. I think that a good starting point for knowing more about the academics of the method is this paper: How Well Sentence Embeddings Capture Meaning. It discusses your method and give some pointers to other papers that also discuss the validity of the method.

Answer 2

TF-IDF is the most simple and starting point of training the embedding for paragraph. SIF and doc2vec provide alternative methods for the embeddings too. Skip thought use encoder to train the embeddings. There are multiple ways of getting the embeddings.

Answer 3

Theoretical 'explanation' of the efficacy of using weighted average of word embedding vectors for certain tasks is provided in the paper A Simple but Tough-to-Beat Baseline for Sentence Embeddings (Combined with an additional step it gives the Smooth Inverse Frequency (SIF). Refer the paper for details.)