2
$\begingroup$

I calculate vector similarities like this:

nlp = spacy.load('en_trf_xlnetbasecased_lg')
a = nlp("car").vector
b = nlp("plant").vector
dot(a, b)/(norm(a)*norm(b))
0.966813

Why are the vector similarities so high for unrelated words for the embedding? This is not the only pair for which they are abnormally high. I also had a similar experience with fastText, so I am wondering, am I misunderstanding something?

Also I am able to get vectors for non-words like "asdfasfdasfd" or "zzz123Y!/§zzzZz", and they differ from each other. How is this possible?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Why are the vector similarities so high for unrelated words for the embedding?

For the specific example you give, I would argue that it makes sense that car and plant have high similarity. This is likely due to phrases such as car manufacturing plant

Also I am able to get vectors for non-words like "asdfasfdasfd" or "zzz123Y!/§zzzZz", and they differ from each other. How is this possible?

For your specific case, since you use the en_trf_xlnetbasecased_lg, the answer is straightforward. Embeddings provided by XLNet are contextual, meaning that even if the word itself isn't a word, you'll get an embedding given the words in its context. Also, it is likely that HuggingFace's implementation uses Byte-Pair Encoding as tokens, making it much more robust to out-of-vocabulary situations.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ This is not the only example pair though. Even if I use complete sentences that do not contain any of the words from the other sentence and are on a different topic, spacy tends to return high similarity scores. Am I missing something fundamentally? Do I have to preprocess differently for this embedding? I also noticed that with the same word with all lower case vs the first letter capitalized, spacy returns a different vector with only ~0.8 cosine similarity. $\endgroup$
    – sandboxj
    Dec 9, 2019 at 11:49
  • 1
    $\begingroup$ Fair enough, there must be weird pairs. Contextual embeddings similarities are typically more difficult to interpret because as opposed to "traditional" embeddings, they may reflect the role of a word in a sentence as opposed to its meaning. I would compare the results you get with XLNet with the ones you get with the normal English model from SpaCy $\endgroup$
    – Valentin Calomme
    Dec 9, 2019 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.