I downloaded the weights for Llama 2 (70B-chat). This process created a folder titled "llama-2-70b-chat," which contained 8 files titled consolidated.00.pth, consolidated.01.pth, and so on until consolidated07.pth. Each file is about 16.84 GB. Here are the names and types of all the tensors in consolidated.00.pth:
- tok_embeddings.weight [32000, 1024]
- norm.weight [8192]
- output.weight [4000, 8192]
- rope.freqs [64]
- layers.0.attention.wq.weight [1024, 8192]
- layers.0.attention.wk.weight [128, 8192]
- layers.0.attention.wv.weight [128, 8192]
- layers.0.attention.wo.weight [8192, 1024]
- layers.0.feed_forward.w1.weight [3584, 8192]
- layers.0.feed_forward.w2.weight [8192, 3584]
- layers.0.feed_forward.w3.weight [3584, 8192]
- layers.0.attention_norm.weight [8192]
- layers.0.ffn_norm.weight [8192]
- layers.1.attention.wq.weight [1024, 8192]
- ... and so on until
- layers.79.ffn_norm.weight [8192]
But why are there 8 separate "consolidated.0X.pt" files? The other 7 files have tensors with the same names and the same shapes, but different values—even the token embeddings have different values!
In fact, if I multiply out the parameter dimensions above and sum them, I get approximately 8.6B parameters, which is far shy of 70B, but almost exactly an eighth of 70B.
I think the answer may relate to how the model uses grouped-query attention; the paper mentions using 8 A100s with tensor parallelism. The params JSON file has this information:
{"dim": 8192, "multiple_of": 4096, "ffn_dim_multiplier": 1.3, "n_heads": 64, "n_kv_heads": 8, "n_layers": 80, "norm_eps": 1e-05, "vocab_size": -1}
Everything else is available publicly in the Llama GitHub repo.
