1
$\begingroup$

I am trying to find the cosine similarity (using glove vector) of two random words. As expected, the distribution of the similarity concentrated around 0 since it is reasonable to think that two random words will not be similar to each other.

However, when I try to do a similar thing to 2 random sets of 10 words, that is I take the average vector of the 10 words in both sets and calculate the cosine similarity, the similarity tends to concentrate at 0.8.

It seems to suggest that given 2 random sentences of 10 words, they are very likely to be similar semantically. What could be the explanation of this?

Included python code to reproduce the result.

import spacy
nlp = spacy.load('en')

vocab = nlp.vocab
words = np.array([x.orth_.encode('utf8') for x in vocab])

hist1 = []
n = 1000
num_words = 1
for _ in range(n):
    x,y = choice(words, size=(2,num_words))
    x = nlp(" ".join([u.decode('utf8') for u in x]))
    y = nlp(" ".join([u.decode('utf8') for u in y]))
    s = x.similarity(y)
    hist1.append(s)

hist10 = []
n = 1000
num_words = 10
for _ in range(n):
    x,y = choice(words, size=(2,num_words))
    x = nlp(" ".join([u.decode('utf8') for u in x]))
    y = nlp(" ".join([u.decode('utf8') for u in y]))
    s = x.similarity(y,)
    hist10.append(s)

plt.hist([hist1,hist10], label=[1,10])
plt.legend()

Distribution of cosine similarity

$\endgroup$
1
  • 1
    $\begingroup$ Interesting observation! You are finding that naive averaging does not yield good document embeddings in the sense that cosine similarities are not centered. I don't know how you would explain why it happens, but I might know a fix: subtract the first principal component, as suggested in A Simple but Tough-to-Beat Baseline for Sentence Embeddings. Welcome to the site! P.S. Are you sure you are averaging and not concatenating the words? $\endgroup$
    – Emre
    Nov 23 '17 at 8:16
1
$\begingroup$

Nothing too surprising here. As you sample more and more words, the sample mean is a better and better estimator of the population mean. This is called the law of large numbers.

$\endgroup$
1
  • $\begingroup$ I am surprised that it converges this rapidly, consider that I only sample 10 words from millions of English words $\endgroup$
    – catethos
    Nov 24 '17 at 8:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.