1
$\begingroup$

The original Transformer paper (Vaswani et al; 2017 NeurIPS) describes the model architecture and the hyperparameters in quite some detail, but it misses to provide the exact (or even rough) model size in terms of parameters (model weights).

I could not find a source with a definite answer on this. Table 3 also mentions a base and a big model, but for none of them model size is given.

How many parameters does a base or a big Transformer model, according to the original implementation by Vaswani et al., have?

$\endgroup$
0

2 Answers 2

1
$\begingroup$

Table 3 has all the values of the hyper-parameters of the models.

See the image below, green are for the base and blue for the big model.

Table 3

You can use these to get the matrices sizes. For example for the multi-headed attention in section 3.2.2

multi headed attention

the matrix $W^{Q}_{i}\in\mathbb{R}^{d_{model}\times d_{k}}$, will have a dimension of $d_{model}\times d_{k} = 512\times 64$ for the base model, which is $32,768$ parameters.

For a single $head_{i}$, that value will be $\times 3 = 98,304$. For the multi-head, that value will be $\times h = \times 8 = 786,432$ parameters for the base model.

You can use the other values from Table 3 to figure out the rest of the model matrices. For examples, $d_{ff}$ is used in section 3.3. Table 3 says that the total of all parts of the model should be $65\times 10^6$ for the base model and $213\times 10^6$ for the big model.

$\endgroup$
1
$\begingroup$

Thanks to Bruno Lubascher's answer I asked ChatGPT and double-checked its answer.

The fixed formula was: $V \times d_{model} + V \times d_{model} + N \times (2 \times h \times 3 \times d_{model} \times d_k + 2 \times d_{model} \times d_{ff})$ for respectively:

  • the token input embeddings
  • the final layer (equivalent to a dot product with token output embeddings)
  • the attention heads
  • the feedforward networks.

Using the base values of the table (plus $V = 37000$ of vocabulary size), I got a total of 60M which was close enough to me to the 65M. Which means that token input/output embeddings make a total of 38M which is 58% of a total of 65M parameters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.