According to this post, the purpose of the multihead is to have 'gradient splitting' across heads, which is achieved by random initialization of weight matrices for Q, K and V in each head. But how can we prove this can solve the problems in using single head?
Specifically, how can the splitting of gradients ensures within each output attention vector for each word it wouldn't overemphasize (the attention) of itself?
In order to see whether one single attention head is enough, we can simply try it. This is precisely what it's done in the article "Are Sixteen Heads Really Better than One?" (publised at NeurIPS'2019). The authors conclude that, for some tasks, having multiple heads is needed especially at training time, while at inference time is it possible to prune a number of heads (depending on the task) without significant performance loss.
