0
$\begingroup$

I'm wondering if there is a way to say something about statistical significance of differences in cosine similarity for word pairs as extracted from a language model. Suppose I have the word pair A-B, and the word pair A-C. I use word2vec, or bert (or whichever other) embeddings, to calculate the cosine similarity for A-B, let's say I get .60. I do the same for A-C, let's say that yields .62. Now, is the difference between .60 and .62 statistically significant? Is that even a reasonable question to ask? I feel it does make sense to ask, and suspect there is something to be said about it. But intuitively, I'd think that I would need to know on how many occurrences the calculation of cosine similarity is based (i.e. how many occurrences of the terms A, B and C there are in the corpus that the embeddings were trained on). Which is something I can't really find out (the corpora used for training the embeddings are not all freely available).

Any ideas on how to say something (statistically) sensible about the difference of .60 and .62 in the above example?

$\endgroup$

1 Answer 1

1
$\begingroup$

Not really, unfortunately.

The interpretation of that difference is going to depend on many factors, such as the objective used in training, the underlying data, the dimensionality of the vectors, whether they're normalized, etc, and this is saying nothing of the difficulty to interpret the values in the first place. Even W2V, arguably conceptually simpler vectors than most with some level of geometric interpretability, would not produce interpretable distinctions at that scale.

$\endgroup$
1
  • $\begingroup$ Thanks much for your answer! $\endgroup$
    – Igor
    Oct 24, 2022 at 7:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.