Actually, the question is all there is

As per the paper I see that the rotations are applied only to the keys and the queries. Why are the rotations not applied to the values as well?

The reasons for applying the rotations to the values as well:

  1. The sinusoidal embeddings are applied to all three: Keys, Queries and, the Values
  2. Why would we not want the embedding values of the tokens to change depending on the relative positions of their occurrences?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.