Could anyone explain to me about the sentence below? What is meant by averaging inhibits it?

Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.

Edit: enter image description here


2 Answers 2


Not sure if this is the right place for this question, but the answer is simple so here we go:

What it says is that multi-head attention allows the model to attend to information from different subspaces in different places. Hence the 'multi' part. It can look in multiple places at the same time. With a single attention head you can only look at one place at the same time. In order to still provide the single attention head with all information, the different places are averaged. Because this averaging occurs, it prevents the model from looking into multiple places (note that this is intentional).

Perhaps a more intuitive example: If you are a government interested in the happiness of your people, and you have a population of say, 20. A method would be to hire 20 happiness inspectors, and assign 1 inspector to each person in your population. < Multi-head attention

However, if you can/want to hire only 1 inspector, they will need to look at some aggregate of the happiness over the whole population to still get an idea of the happiness score. Thus, the 20 scores are averaged, and given to the inspector (let's skip how we get these scores in this example). The inspector sees only the average, and thus knows nothing about individual happiness scores > the averaging prevents - or inhibits - the inspector from seeing the individual scores. < Single attention head

(not specifically an expert in the field of attention mechanisms, so forgive me some minor technical mistakes)

  • $\begingroup$ Hi Tim, thanks for the reply. I have edited my original question by attaching a new picture. From the analogy u provided, it does make sense, but I wonder where does the averaging occurs in the single head attention mechanism. $\endgroup$
    – Jayden
    Commented Jun 21, 2022 at 14:56
  • $\begingroup$ I’m afraid that’s a completely different question: “Where does averaging occur in single-head attention mechanism X?” Like I said, I’m no expert on attention mechanisms, so I cannot answer your follow-up question without some in-depth research. But then it’s probably more efficient if you did that yourself. $\endgroup$
    – Tim J
    Commented Jun 22, 2022 at 21:16

Not an expert on transformers, but here is my understanding.

Firstly, to compare multi-head and single-head attention, they should be of the same dimensionality. Let the input to each self-attention of the multi-head case be $(\tilde Q_i,\tilde K_i,\tilde V_i)$, and let the input to the self-attention of the single-head case be $(\tilde Q,\tilde K,\tilde V)$, then the dimension of $\tilde Q$ should be identical to that of $(\tilde Q_1,\dots,\tilde Q_h)$ where $h$ is the number of heads. The same applies to $K$ and $V$.

Define the above notation as: $\tilde Q_i = Q W_i^Q$. Let $S(\cdot) = \operatorname{softmax}(\cdot / \sigma)$ be the scaled softmax operator (the scaling factor is omitted for convenience below). By multi-head attention formula, the output is the concatenation of $S(\tilde Q_i\tilde K_i^\top)\tilde V_i$ followed by the output projection. This achieves:

Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.

so each $S(\tilde Q_i\tilde K_i^\top)$ is a way to attend to information. On the other hand, however, by single-head attention, the output is

$$ \begin{aligned} S(\tilde Q\tilde K^\top)\tilde V &= S\left(\begin{pmatrix}\tilde Q_1 & \dots & \tilde Q_h\end{pmatrix}\begin{pmatrix}\tilde K_1^\top\\ \vdots \\ \tilde K_h^\top\end{pmatrix}\right)\tilde V\\ &= S\left(\sum_{i=1}^h\tilde Q_i\tilde K_i^\top\right)\tilde V\\ \end{aligned} $$

followed by the output projection. There's only one set of softmax attention weights now; thus the model is able to attend to information in one manner only.

The key point is: notice that there's a summation single-head attention formula. But summation is equivalent to averaging up to a constant. Hence, we may sufficiently state that:

averaging inhibits this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.