As far as I understand and looked into Attention Is All You Need and Transformer model for language understanding, the Self Attention at Scaled Dot-Product Attention is calculating $query$ and $key$ of the same word, which is the diagonal of the matrix in the diagram.

The $q \cdot k$ of the same word will generate the largest value, which means that a word attends to itself in a sentence. Is it correct or am I missing something?

Why Self Attention does not exclude the $q \cdot k$ of the same word itself?

enter image description here


1 Answer 1


It is not necessarily the case. The matrics $K$ and $Q$ can be very different. The intuition is that these two projections allow the model to search for a particular piece of information in the hidden states. From that perspective, there is no need to exclude the query state.

If you look at analyses of what the attention heads in trained models do, you can see that most of the patterns are not diagonal as you would expect. Some of them look diagonal, some really are, but it is often a shifted diagonal.

Example from Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (Voita et al., ACL 2019):

Self-attention examples, Voita et al.

Another example from From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions (Mareček & Rosa, 2019)

Self-attention examples, Mareček and Rosa

  • $\begingroup$ Thanks so much for pointing to the papers. Much appreciated. $\endgroup$
    – mon
    Jul 22, 2022 at 4:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.