I think the cosine similarity of negative values has its own meaning.
If you softmax the cosine similarity of Q and K, wouldn't it prevent Transformer from using information with the opposite meaning?
1
-
$\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$– Community BotMar 24 at 6:51
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
Consider that before the attention block, you apply a (trainable) linear projection, and that there are many attention heads.
This gives the needed degrees of freedom to handle the potential logic of negative cosine similarity that you think the model is lacking. For instance, one head might learn to rotate the query while another one might learn not to rotate it.
-
$\begingroup$ If we make good use of the cosine similarity of negative values, can't we get the information that we can get from two heads from one head? $\endgroup$ Mar 24 at 10:50
-
$\begingroup$ Maybe, but that would be a different model. You can experiment with that approach. $\endgroup$– noeMar 24 at 11:02
-
$\begingroup$ Please, consider accepting the answer (with the tick ✓ next to it). $\endgroup$– noeMar 24 at 11:03
-
$\begingroup$ In fact, I posted it on Stack Overflow for the first time. I appreciate you for replying. @noe $\endgroup$ Mar 24 at 11:56
-
$\begingroup$ At relu activation, cosine similarities with values below a certain value are likely to be treated the same @noe $\endgroup$ Mar 24 at 12:50