Why does averaging a sentence's worth of word vectors work?

Question

I am working on a text classification problem using r8-train-all-terms.txt, r8-test-all-terms.txt from https://www.cs.umb.edu/~smimarog/textmining/datasets/. The goal is to predict the label using a Random Forest classifier.

Each text sentence as been vectorized using the GoogleNews word vectors. The embedding source can be found here: https://github.com/mmihaltz/word2vec-GoogleNews-vectors

In the example I am following along with there is one step that irks me - there is a step that converts my array of vectorized tokens to a single vector by taking the mean over the tokens e.g.

def transform(self, data):
        v = self.word_vectors.get_vector('king')
        self.D = v.shape[0]

        X = np.zeros((len(data), self.D))
        n = 0
        emptycount = 0  
        for sentence in data:
            tokens = sentence.split()
            vecs = []
            m = 0
            for word in tokens:
                try:
                    vec = self.word_vectors.get_vector(word)
                    vecs.append(vec)
                    m += 1
                except KeyError:
                    pass
            if len(vecs) > 0:
                vecs = np.array(vecs)
                X[n] = vecs.mean(axis=0)    # take the mean of the vectors?  what does it mean?
            else:
                emptycount += 1
            n += 1
        print("Number of samples with no words found: %s / %s" % (emptycount, len(data)))
        return X

I am leaving out some boilerplate but later on I run the model and the results are surprisingly good:

model = RandomForestClassifier(n_estimators = 200)
model.fit(XTrain, YTrain)
print("train score:", model.score(XTrain, YTrain))
print("test score:", model.score(XTest, YTest))

> train score: 0.9992707383773929
> test score: 0.9378711740520785

I understand that the random forest model expects to have one row per example so it is unable to consume a sequence of embeddings like a RNN might. So you are required to convert to single row (1-D array).

My question is: WHY does it work? It seems at odds to me that the averaged word vectors would be able to capture anything about the context or meaning of a sentence by merely averaging over the encodings.

Best case scenario I would expect this technique breaks down for larger blocks of text because you would tend to squash all your examples into the same neighborhood of your input space.

It would be great to get some clarification on this.

Answer 1

It works for the same reason why the good old bag-of-words + TF-IDF works. Despite loosing some word ordering information, a text can be still classified by the typical keywords. Since texts on different topics differ a lot with respect to the vocabulary used, simply putting together the words' embeddings might work surprisingly well.

Here is a paper that shows that a simple sentence embedding methods beats sophisticated supervised methods including RNNs and LSTMs. Their method is just a weighted average of the word vectors, modified a bit using PCA/SVD. Section 4.3 tells that the word ordering plays a role, but not too much.