I'm puzzling to understand why the method of averaging word embeddings works in order to obtain sentence embedding, in particular considering the exercize of this post How to obtain vector representation of phrases using the embedding layer and do PCA with it. My current question actually is to understand the theory behind that more practical post.
The answer to the question linked uses a method for sentence embedding that is averaging the word embeddings (in the most naive and simplest case in which we obtain the word embeddings by extracting vectors from the embedding layer of the neural network model and so without using pretrained NN model).
This method seems to work because the words in the PCA space make clusters according to the class labels they belong to. Why does it work ?
My personal explanation that I have tried to give is the following: the Tokenizer assigns an integer to each word (let us assume that we choose words as tokens) on the basis of word frequency: lower integer means more frequent word (often the first few are stop words because they appear a lot) (from What does Keras Tokenizer method exactly do?).
Now sentences are vectors of integers. After padding and truncatin, we can feed them to our NN.
In the training phase, the Neural Network receives in input a series of symbols: first the symbols of the first sentence and assigns to each of them a vector of weights (of which we can decide the dimension). Then it looks which is the label and through the backpropagation algorithm it updates its weights on the basis of this label. Sentences belonging to the same class will have weights more similar one respect to each other but also the NN captures the structure of the sentences (this last observation is from the answer to the post How does Keras 'Embedding' layer work?) and so maybe sentences that share more words in common will have weights vectors more similar (I tested myself this by doing an other simple exercize of classification where I considered as input data a set of words, I chose letters as tokens and I considered as word embedding the average of the embeddings of all the letters in the word. In the PCA plot, besides clustering according to the classes, words that share more letters in common were closer with respect to those that differ more).
And so, making the average of the words embeddings vectors to obtain the sentence embedding vectors comes out to be a good method because the average is like a "weighted average" since the weights vectors are assigned to each symbol based on the class of the entire set of symbols (i.e. sentence) and also based on the general structure of the set of symbols (i.e. frequency of words) ?
I did not find a research paper that used this method (they very often use more sophisticated methods that I have no time to try in this moment). But I found this answer to this other post word2vec - what is best? add, concatenate or average word vectors?. Here it links to a lesson in which a Professor explains that averaging word vectors works incredibly well to capture all the statistics.