1
$\begingroup$

In natural language processing, the cosine similarity is often used to compute the similarity between two words. It is bounded between [-1, 1]. Supposedly, 1 means complete similarity, -1 means something like antonyms, and 0 means no relationship between the words, although I am unsure whether that fully holds true in praxis. For another application, I need to convert the cosine similarity to a probability between 0 and 1. A straightforward solution would be to take the absolute value of the cosine similarity, but does this make sense? My goal is simply to assign higher scores to words that occur in similar contexts (i.e. could be swapped out and still leave the sentence plausible).

$\endgroup$
1
  • $\begingroup$ You can try using the sigmoid function. torch.nn.Sigmoid $\endgroup$
    – Take Lama
    Jun 14, 2023 at 12:15

2 Answers 2

2
$\begingroup$

You can try to normalize the similarity: norm_sim = (sim + 1) / 2, where sim is the cosine.

In this case, 0 means opposite similarity, 0.5 means no relationship between words and 1 means complete similarity.

In my opinion, taking the absolute value of the cosine wouldn't have much sense because a word that has complete similarity with another one gets the same value as its antonym.

source

If you want to assign an higher score to antonyms than to unrelated words, you could try assigning weights to the cosine values < 0 that reduce more the value as it approaches 0.

For example:

def weights(x):
  if x < 0:
   return ((0.75*x)+1)/2
  else:
   return (x+1)/2

I know this may be an awful choice of weights, but i think the idea is valid

$\endgroup$
3
  • $\begingroup$ Thank you for your answer. I thought about rescaling like this, but my issue is with assigning antonyms a lower score than words with no relationship. That way, when I want to swap out words for similar words, I may disregard antonyms that still make a sentence meaningful in favour of words with no relation. It seems my issue then may be more with cosine similarity than how to normalise it... $\endgroup$ Mar 17, 2023 at 21:06
  • $\begingroup$ Hi, I edited the answer $\endgroup$
    – Iya Lee
    Mar 17, 2023 at 21:25
  • 1
    $\begingroup$ Thanks, that's a creative idea. I might try it, although choosing arbitrary weights seems a bit hacky. But if it works, so what? $\endgroup$ Mar 18, 2023 at 8:20
0
$\begingroup$

I'm not sure there's a right answer here. It depends on what probability distribution you're trying to model.

If you consider a unit circle and want to model the probability distribution of selecting a random angle uniformly, you'd want to use $$ p(x) = \frac{\pi - acos(x)}{\pi} $$ where x is the cosine similarity score.

However, if you want to model the probability distribution of selecting a cosine value uniformly, you'd want to use

$$ p(x) = \frac{x - (-1)}{1 - (-1)} = \frac{x+1}{2} $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.