In transformer architecture multi head attention blocks are used. While visualizing their output it can be seen that every layer has learnt different relations of words. e.g., layer 5 has learnt that "It" is more related to "animal".

sample attention layer 5

Question here is, when all attention layers are running in parallel, what is different fed to different layer so that they learn different things?
Note: this answer is not clear - why-and-how-bert-can-learn-different-attentions-for-each-head


1 Answer 1


All heads are fed the exact same input. Each head learns different weight values because:

  • The attention heads, along with the rest of the network, are initialized randomly.
  • The back-propagated gradients each head receives are different. This is because the result of the multi-head attention is the concatenation of each head. When back-propagating through the concatenation, the gradient is split among the heads, so that each head gets a piece of the back-propagated gradient and, therefore, the gradient information received by each head is different.
  • $\begingroup$ thanks for reply... How "The back-propagated gradients each head receives are different" $\endgroup$ Jun 7, 2021 at 16:27
  • $\begingroup$ I added a clarification to the answer with the doubt you pointed out. $\endgroup$
    – noe
    Jun 7, 2021 at 17:12
  • $\begingroup$ Please, consider upvoting the answer if it was useful, and also marking it as correct if deemed so. Otherwise, please describe what is no clear from the answer or why you think it is not correct. $\endgroup$
    – noe
    Jun 12, 2021 at 5:32
  • $\begingroup$ Surely will mark as answer.. I am still working on overall piece $\endgroup$ Jun 12, 2021 at 7:10
  • $\begingroup$ @SandeepBhutani was the answer helpful? Did it clarify your doubts? $\endgroup$
    – noe
    Jul 15, 2021 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.