I am working on a text classification problem using r8-train-all-terms.txt, r8-test-all-terms.txt from https://www.cs.umb.edu/~smimarog/textmining/datasets/. The goal is to predict the label using a Random Forest classifier.
Each text sentence as been vectorized using the GoogleNews word vectors. The embedding source can be found here: https://github.com/mmihaltz/word2vec-GoogleNews-vectors
In the example I am following along with there is one step that irks me - there is a step that converts my array of vectorized tokens to a single vector by taking the mean over the tokens e.g.
def transform(self, data): v = self.word_vectors.get_vector('king') self.D = v.shape X = np.zeros((len(data), self.D)) n = 0 emptycount = 0 for sentence in data: tokens = sentence.split() vecs =  m = 0 for word in tokens: try: vec = self.word_vectors.get_vector(word) vecs.append(vec) m += 1 except KeyError: pass if len(vecs) > 0: vecs = np.array(vecs) X[n] = vecs.mean(axis=0) # take the mean of the vectors? what does it mean? else: emptycount += 1 n += 1 print("Number of samples with no words found: %s / %s" % (emptycount, len(data))) return X
I am leaving out some boilerplate but later on I run the model and the results are surprisingly good:
model = RandomForestClassifier(n_estimators = 200) model.fit(XTrain, YTrain) print("train score:", model.score(XTrain, YTrain)) print("test score:", model.score(XTest, YTest)) > train score: 0.9992707383773929 > test score: 0.9378711740520785
I understand that the random forest model expects to have one row per example so it is unable to consume a sequence of embeddings like a RNN might. So you are required to convert to single row (1-D array).
My question is: WHY does it work? It seems at odds to me that the averaged word vectors would be able to capture anything about the context or meaning of a sentence by merely averaging over the encodings.
Best case scenario I would expect this technique breaks down for larger blocks of text because you would tend to squash all your examples into the same neighborhood of your input space.
It would be great to get some clarification on this.