# Vision Transformer ViT Parameter count

The Vision Transformer paper An Image is with 16x16 words by Dosovitskiy et al. (2021) includes the following table:

Can someone explain how they get the parameter counts or where my calculation is wrong? Let's look at ViT-Base: Each attention layer requires three $$768 \times 768$$ matrixes to produces $$Q, K, V$$ from the input. Then the result of each attention layer is concatenated and transformed back to $$D$$ requiring another $$(12 \cdot 768) \times 768$$ matrix.

With 12 heads this adds up to $$12 \cdot 768 \cdot 768 + 12 \cdot 768 \cdot 768 \approx 14M$$ parameter per MSA head. And we add the parameters for the MLP ($$2 * 768*3072 \approx 4.7M$$).

Using 12 layers this would imply $$12 \cdot (14 + 4.7) \approx 224M$$ parameter instead of the 86M specified?

My calculation was based on a wrong understanding of the self attention mechanism. In Attention is all you need the authors point out that they won't use the full $$768 \times 768$$ matrices when they make use of multi-head attention but rather use $$768 / h$$ as the internal dimension where $$h$$ is the number of heads.