Question partially inspired by this post about the need of multi-head attention mechanism.

For me though it is still not clear how we will be able to initialise those attention heads in a diverse way(so that they potentially can - as stated in the Attention is all you need paper - attend to information from different representation subspaces at different positions) and most importantly preserve this diversity during the training process.


1 Answer 1


I think, what you are looking for is $d_k = d_v = d_{model}/h$ [1] where $h$ number of heads and $d_{model}$ dimensions of keys, values and queries for single attention version of the model. In relation of the model architecture and its embedding specifically, the above translates to

$Query Size = Embedding Size / h$

Model input

According to the above, at the input embedding layer weights for each head are stacked together in the single embedding matrix. Stacking together, hmm.. (stacking, ensembling).

by Ketan Doshi

Per head scores

As in the normal self-attention, attention score is computed per head but given the above, these operations also take in place as a single matrix operation and not in a loop. The scaled dot product along with other calculations take place here.

Multi head merge

As a final step, the attention score of each head is merged by simply reshaping the full attention score matrix so that the per-head attention scores are concatenated into a single attention score.

by Ketan Doshi

Summing it up

With a clearer view of what the architecture of the model is and how its computational graphs looks like, we can go back to your original questions and say:

The multi-headed model can capture richer interpretations because the embedding vectors for the input gets "segmented" across multiple heads and therefore different sections of the embedding can attend different per-head subspaces that link back to each word.

In a more general sense, one can argue that running through the scaled dot-product attention multiple times in parallel and concatenating is a form of ensembling.

I hope it helps. For more details about the several operation taking place in the transformer architecture, I suggest you have a look at this post from which my answer is heavily influenced from: https://towardsdatascience.com/transformers-explained-visually-part-3-multi-head-attention-deep-dive-1c1ff1024853

[1] 3.2.2 https://arxiv.org/pdf/1706.03762.pdf

  • $\begingroup$ Thanks! I think it is the right one. As just a follow up thoughts I wonder if it is harmless to split embeddings into segments. If I'm not mistaken it is as if we are assuming that subspaces convey different meanings and it is probably true but ideal boundaries are unlikely to be at the split cuts. I wonder if it is beneficial to randomly sample embedding space and form segments that can even overlap. Feels like it would give us a higher chance to find interesting relationships. It would also remove this requirement for number of heads to be divisible for the number of embedding dimensions. $\endgroup$
    – Deil
    May 25, 2021 at 16:18
  • $\begingroup$ nw hope it helped! I am not sure I fully understand what you mean but, while it sounds entirely reasonable it is in essence what mulihead is doing during training. $\endgroup$
    – hH1sG0n3
    May 26, 2021 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.