# Difficulty interpreting word embedding vector similarity (spaCy)

I calculate vector similarities like this:

nlp = spacy.load('en_trf_xlnetbasecased_lg')
a = nlp("car").vector
b = nlp("plant").vector
dot(a, b)/(norm(a)*norm(b))
0.966813


Why are the vector similarities so high for unrelated words for the embedding? This is not the only pair for which they are abnormally high. I also had a similar experience with fastText, so I am wondering, am I misunderstanding something?

Also I am able to get vectors for non-words like "asdfasfdasfd" or "zzz123Y!/§zzzZz", and they differ from each other. How is this possible?

Why are the vector similarities so high for unrelated words for the embedding?

For the specific example you give, I would argue that it makes sense that car and plant have high similarity. This is likely due to phrases such as car manufacturing plant

Also I am able to get vectors for non-words like "asdfasfdasfd" or "zzz123Y!/§zzzZz", and they differ from each other. How is this possible?

For your specific case, since you use the en_trf_xlnetbasecased_lg, the answer is straightforward. Embeddings provided by XLNet are contextual, meaning that even if the word itself isn't a word, you'll get an embedding given the words in its context. Also, it is likely that HuggingFace's implementation uses Byte-Pair Encoding as tokens, making it much more robust to out-of-vocabulary situations.

• This is not the only example pair though. Even if I use complete sentences that do not contain any of the words from the other sentence and are on a different topic, spacy tends to return high similarity scores. Am I missing something fundamentally? Do I have to preprocess differently for this embedding? I also noticed that with the same word with all lower case vs the first letter capitalized, spacy returns a different vector with only ~0.8 cosine similarity. – sandboxj Dec 9 '19 at 11:49
• Fair enough, there must be weird pairs. Contextual embeddings similarities are typically more difficult to interpret because as opposed to "traditional" embeddings, they may reflect the role of a word in a sentence as opposed to its meaning. I would compare the results you get with XLNet with the ones you get with the normal English model from SpaCy – Valentin Calomme Dec 9 '19 at 11:51