I am trying to find the cosine similarity (using glove vector) of two random words. As expected, the distribution of the similarity concentrated around 0 since it is reasonable to think that two random words will not be similar to each other.
However, when I try to do a similar thing to 2 random sets of 10 words, that is I take the average vector of the 10 words in both sets and calculate the cosine similarity, the similarity tends to concentrate at 0.8.
It seems to suggest that given 2 random sentences of 10 words, they are very likely to be similar semantically. What could be the explanation of this?
Included python code to reproduce the result.
import spacy
nlp = spacy.load('en')
vocab = nlp.vocab
words = np.array([x.orth_.encode('utf8') for x in vocab])
hist1 = []
n = 1000
num_words = 1
for _ in range(n):
x,y = choice(words, size=(2,num_words))
x = nlp(" ".join([u.decode('utf8') for u in x]))
y = nlp(" ".join([u.decode('utf8') for u in y]))
s = x.similarity(y)
hist1.append(s)
hist10 = []
n = 1000
num_words = 10
for _ in range(n):
x,y = choice(words, size=(2,num_words))
x = nlp(" ".join([u.decode('utf8') for u in x]))
y = nlp(" ".join([u.decode('utf8') for u in y]))
s = x.similarity(y,)
hist10.append(s)
plt.hist([hist1,hist10], label=[1,10])
plt.legend()
