Cosine similarity of averaged random word vectors

I am trying to find the cosine similarity (using glove vector) of two random words. As expected, the distribution of the similarity concentrated around 0 since it is reasonable to think that two random words will not be similar to each other.

However, when I try to do a similar thing to 2 random sets of 10 words, that is I take the average vector of the 10 words in both sets and calculate the cosine similarity, the similarity tends to concentrate at 0.8.

It seems to suggest that given 2 random sentences of 10 words, they are very likely to be similar semantically. What could be the explanation of this?

Included python code to reproduce the result.

import spacy

vocab = nlp.vocab
words = np.array([x.orth_.encode('utf8') for x in vocab])

hist1 = []
n = 1000
num_words = 1
for _ in range(n):
x,y = choice(words, size=(2,num_words))
x = nlp(" ".join([u.decode('utf8') for u in x]))
y = nlp(" ".join([u.decode('utf8') for u in y]))
s = x.similarity(y)
hist1.append(s)

hist10 = []
n = 1000
num_words = 10
for _ in range(n):
x,y = choice(words, size=(2,num_words))
x = nlp(" ".join([u.decode('utf8') for u in x]))
y = nlp(" ".join([u.decode('utf8') for u in y]))
s = x.similarity(y,)
hist10.append(s)

plt.hist([hist1,hist10], label=[1,10])
plt.legend()


• Interesting observation! You are finding that naive averaging does not yield good document embeddings in the sense that cosine similarities are not centered. I don't know how you would explain why it happens, but I might know a fix: subtract the first principal component, as suggested in A Simple but Tough-to-Beat Baseline for Sentence Embeddings. Welcome to the site! P.S. Are you sure you are averaging and not concatenating the words?
– Emre
Nov 23 '17 at 8:16