Reason: Average word vector embedding encodes word content and word order effectively

Question

I was going through a research paper: FINE-GRAINED ANALYSIS OF SENTENCE EMBEDDINGS USING AUXILIARY PREDICTION TASKS

The key take away was Comparison of Encoder decoder and average word sentence embedding validated for accuracy for sentence embedding on 3 basic language characteristics- sentence length, word content, and word order.

I found it surprising, that an averaged word embedding for a sentence is better at predicting presence of a word in a sentence than an Encoder Decoder. Also, how is it that, increasing embedding size deteriorates it's performance.

Same question goes for word ordering, how is average word embedding able to do that? The experiments are able to explain, what would happen if prediction is based on permutation of words, but the explanation doesn't feel intuitive to me. How is simple avg word embedding able to contain information like word order when taking it's average kinda nullifies the order info

Answer 1

When reading the paper there actually is some points towards the background of these phenomena:

About predicting a word meaning there is only the surprise present, but about predicting the order there was a whole section about the statistical influence of natural language underlying mechanics to the performance of the CBOW. It appears, that order of sentences or even word pairs was easy for CBOW whereas a random permutation with no natural order dropped also the performance.

If we go very basics, CBOW (Continuous Basket Of Words) mission is to:

Predicting a word given its context. [1]

Thus, with even an average (or only because of it) we can identify which word is which, although in a little bit weird way around. Just because that is how it is behaving. Secondly, basket is for training the neural network, not the full algorithm. What is the model obtained would be different of the way it gets its data.

What makes me guessing is the drop on ability to identify from larger range of words. As we all are surprised how averages make the trick so, well, maybe the true nature is revealed with larger dimensions.

[1] https://www.quora.com/What-are-the-continuous-bag-of-words-and-skip-gram-architectures