Actually, the question is all there is
As per the paper I see that the rotations are applied only to the keys and the queries. Why are the rotations not applied to the values as well?
The reasons for applying the rotations to the values as well:
- The sinusoidal embeddings are applied to all three: Keys, Queries and, the Values
- Why would we not want the embedding values of the tokens to change depending on the relative positions of their occurrences?