# How many parameters does the vanilla Transformer have?

The original Transformer paper (Vaswani et al; 2017 NeurIPS) describes the model architecture and the hyperparameters in quite some detail, but it misses to provide the exact (or even rough) model size in terms of parameters (model weights).

I could not find a source with a definite answer on this. Table 3 also mentions a base and a big model, but for none of them model size is given.

How many parameters does a base or a big Transformer model, according to the original implementation by Vaswani et al., have?

Table 3 has all the values of the hyper-parameters of the models.

See the image below, green are for the base and blue for the big model.

You can use these to get the matrices sizes. For example for the multi-headed attention in section 3.2.2

the matrix $$W^{Q}_{i}\in\mathbb{R}^{d_{model}\times d_{k}}$$, will have a dimension of $$d_{model}\times d_{k} = 512\times 64$$ for the base model, which is $$32,768$$ parameters.

For a single $$head_{i}$$, that value will be $$\times 3 = 98,304$$. For the multi-head, that value will be $$\times h = \times 8 = 786,432$$ parameters for the base model.

You can use the other values from Table 3 to figure out the rest of the model matrices. For examples, $$d_{ff}$$ is used in section 3.3. Table 3 says that the total of all parts of the model should be $$65\times 10^6$$ for the base model and $$213\times 10^6$$ for the big model.

The fixed formula was: $$V \times d_{model} + V \times d_{model} + N \times (2 \times h \times 3 \times d_{model} \times d_k + 2 \times d_{model} \times d_{ff})$$ for respectively:
Using the base values of the table (plus $$V = 37000$$ of vocabulary size), I got a total of 60M which was close enough to me to the 65M. Which means that token input/output embeddings make a total of 38M which is 58% of a total of 65M parameters.