3
$\begingroup$

I am comparing words in HuggingFace web UI using e5-small-v2, one of the best vector embedding models:

enter image description here

Assuming that the scores are in the range from 0 to 1, how come all the scores are so high? In fact, I was not able to produce any example with a score below 0.7. Is there something basic I am missing about vector embeddings?

$\endgroup$

1 Answer 1

2
$\begingroup$

I explored this very problem in one of my medium posts.Why are Cosine similarities almost always positive.

Quoting from that:

In other words, the cosine similarity has a positive contribution if the corresponding dimensions in the vector are either both pointing in the positive direction, or both in the negative direction. Since the max pool moves some dimensions in the positive direction, it is statistically imperative in a large dimensional space to have a net positive bias.

If this causes issues for processes downstream, you can train a transformation matrix that multiplies each of the two vectors. This is done such that the cosine distances of dissimilar vectors get larger to span the full -1 to +1 range. The matrix would have to be trained on a subset of data that closely resembles your intended application.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.