# How exactly have T5 3B and 11B models been scaled?

I've been trying to understand the difference between the large T5 model, and the 3B model. I thought it was a typo in a paper referencing it, but the original paper says the same thing. (For reference, I'm quoting from https://arxiv.org/pdf/1910.10683.pdf#page=36)

Base uses dmodel of 768, 12 heads and dkv of 64. 12 x 64 = 768, all good. 12 layers in each of encoder and decoder. dff is 3072, which means the FFN, in PyTorch terms would be:

x = nn.Linear(768, 768 * 4)
x = F.relu(x)
x = nn.Linear(768 * 4, 768)


We then have Large size, which increases the model dimension, dmodel, to 1024, with 16 heads each of dkv=64, 16 x 64 = 1024; it keeps the same x4 ratio for the FFN, so it is 1024 -> 4096 -> 1024. It uses 24 layers in each of encoder and decoder, and has 770M params, compared to the base model 220M params.

3B and 11B... we use dmodel = 1024, a 24 layer encoder and decoder, and dkv = 128. For the “3B” variant, we use dff = 16,384 with 32-headed attention

Maybe it is just sloppy language, but I took 32-headed attention to mean there are 32 heads? But that is twice as many heads and twice the dkv, 32 x 128 = 4096 which is not equal to the dmodel of 1024. Though 4096 would then make sense, as then the standard 4x in the FFN would match the dff they give us: 4096 -> 16384 -> 4096.

However, I believe that would be a lot more than 3B parameters then? So did they mean to write "with head dimension of 32" ? And then their FFN is doing a fairly novel 16x ratio: 1024 -> 16384 -> 1024.

The paper goes on to say:

for “11B” we use dff = 65,536 with 128-headed attention...

So does this mean their model dimension of 1024 is divided into 128 heads each of dimension dkv = 8? And then their FFN is 1024 -> 65536 -> 1024?

If this is true, does anyone have any insight, papers, references to experiments, etc. on why if they were sticking with a model dimension of 1024, they didn't just stick with 16 heads of dkv=64?

By the way, not increasing model dimension as scaling up is unusual (e.g. GPT-3, https://arxiv.org/pdf/2005.14165.pdf#page=8, keeps increasing it, and keeps head-dim almost the same; they also always use dff = dmodel x 4), so I've been surprised the paper doesn't draw more attention to it, only saying it is efficient on a TPU. The various blogs and articles about T5 that I've found don't seem to mention it at all.