Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am comparing words in HuggingFace web UI using e5-small-v2, one of the best vector embedding models:
Assuming that the scores are in the range from 0 to 1, how come all the scores are so high? In fact, I was not able to produce any example with a score below 0.7. Is there something basic I am missing about vector embeddings?
I explored this very problem in one of my medium posts.Why are Cosine similarities almost always positive.
Quoting from that:
In other words, the cosine similarity has a positive contribution if the corresponding dimensions in the vector are either both pointing in the positive direction, or both in the negative direction. Since the max pool moves some dimensions in the positive direction, it is statistically imperative in a large dimensional space to have a net positive bias.
If this causes issues for processes downstream, you can train a transformation matrix that multiplies each of the two vectors. This is done such that the cosine distances of dissimilar vectors get larger to span the full -1 to +1 range. The matrix would have to be trained on a subset of data that closely resembles your intended application.
