In transformer architecture multi head attention blocks are used. While visualizing their output it can be seen that every layer has learnt different relations of words. e.g., layer 5 has learnt that "It" is more related to "animal".
Question here is, when all attention layers are running in parallel, what is different fed to different layer so that they learn different things?
Note: this answer is not clear - why-and-how-bert-can-learn-different-attentions-for-each-head