Why does averaging word embedding vectors (exctracted from the NN embedding layer) work to represent sentences?

Question

I'm puzzling to understand why the method of averaging word embeddings works in order to obtain sentence embedding, in particular considering the exercize of this post How to obtain vector representation of phrases using the embedding layer and do PCA with it. My current question actually is to understand the theory behind that more practical post.

The answer to the question linked uses a method for sentence embedding that is averaging the word embeddings (in the most naive and simplest case in which we obtain the word embeddings by extracting vectors from the embedding layer of the neural network model and so without using pretrained NN model).

This method seems to work because the words in the PCA space make clusters according to the class labels they belong to. Why does it work ?

My personal explanation that I have tried to give is the following: the Tokenizer assigns an integer to each word (let us assume that we choose words as tokens) on the basis of word frequency: lower integer means more frequent word (often the first few are stop words because they appear a lot) (from What does Keras Tokenizer method exactly do?).

Now sentences are vectors of integers. After padding and truncatin, we can feed them to our NN.

In the training phase, the Neural Network receives in input a series of symbols: first the symbols of the first sentence and assigns to each of them a vector of weights (of which we can decide the dimension). Then it looks which is the label and through the backpropagation algorithm it updates its weights on the basis of this label. Sentences belonging to the same class will have weights more similar one respect to each other but also the NN captures the structure of the sentences (this last observation is from the answer to the post How does Keras 'Embedding' layer work?) and so maybe sentences that share more words in common will have weights vectors more similar (I tested myself this by doing an other simple exercize of classification where I considered as input data a set of words, I chose letters as tokens and I considered as word embedding the average of the embeddings of all the letters in the word. In the PCA plot, besides clustering according to the classes, words that share more letters in common were closer with respect to those that differ more).

And so, making the average of the words embeddings vectors to obtain the sentence embedding vectors comes out to be a good method because the average is like a "weighted average" since the weights vectors are assigned to each symbol based on the class of the entire set of symbols (i.e. sentence) and also based on the general structure of the set of symbols (i.e. frequency of words) ?

I did not find a research paper that used this method (they very often use more sophisticated methods that I have no time to try in this moment). But I found this answer to this other post word2vec - what is best? add, concatenate or average word vectors?. Here it links to a lesson in which a Professor explains that averaging word vectors works incredibly well to capture all the statistics.

Answer 1

A simple, intuitive explanation- think of each latent dimension as a measure of some (very abstract) quality or property of a word. The value a word's coordinate has in that dimension describes how aligned or opposed that word is to that property. Words have a higher similarity when they are aligned with the same abstract concepts and opposed to the same ones.

When you average the coordinates over the words in a sentence, what happens to the coordinates that many words agree upon? In a very, very simple example of a two word sentence and a two dimensional embedding, say the first word has the embedding $(1, -1)$ and the second $(1, 1)$. The average of the two is $(1, 0)$. Since the words agree on the first dimension, the average is large. Since they disagree on the second, they cancel each other out. Adding in a third word with embedding $(1,0)$ results in the same average. All 3 words are positively associated with the first abstract concept. For the second abstract concept, one word is positively associated, one is negatively, and the last is neutral. These all combine to be neutral. So the embedding average in a sense captures the associations that words agree on, and when words disagree the average becomes more neutral to the concept.

For tasks where the ordering of the words is not important but the collective associations are, this is very convenient- it allows representing sentences or collections of words of arbitrary length in the same exact way as individual words, and the embeddings are directly comparable. Two sentences are similar when the average of the similarities between the individual words is large. Take a sentence with words $(x_1, x_2)$ and one with $(y_1, y_2)$. Computing their average embeddings followed by the dot product of the embeddings can be written as

$$ <\frac{x_1 + x_2}{2},\frac{y_1+ y_2}{2}>$$

But since inner products are linear, we can rewrite this as

$$ \frac{1}{4}(<x_1, y_1> + <x_1, y_2> + <x_2, y_1> + <x_2, y_2>)$$

Which shows that sentence similarity calculated in this way is equivalent to the average similarity when selecting a word in the first sentence and one from the second. This gives another way to think about the averaged embeddings in a way that directly relates to the similarities between the individual words.

An interesting and important connection pops up when we consider the distance between the embeddings- this takes into account the pairwise similarity between words within the sentences as well as between. This becomes equivalent to something called Maximum Mean Discrepancy which has a great number of important properties when examining whether two samples came from the same distribution or not.

Answer 2

This sentence representation works well for some tasks (e.g. document classification, topic modeling) and poorly for others (e.g. sequence modeling).

The reason it works well for, say, document classification is because distance is semantically meaningful in the embedding space. If the embeddings are trained well, then a word vector's location in the embedding space (roughly) corresponds to that word's meaning. If you take the average of all word vectors in a sentence, the result is a vector which represents the average meaning of the sentence.

For example, the sentence "I have dogs because I like dogs" will probably have an average embedding near the word "dog". And the sentence "I have cats because I like cats" will have an average embedding closer to "cat". The average embedding gives you a decent idea of the sentence's topic.