All Questions
Tagged with scalability gpt
1 question
1
vote
1
answer
351
views
How does compute required scale with number of model parameters?
GPT-3 has 175 billion parameters, required ~$3.114 * 10^{23}$ FLOPS, and took approximately one month to train on ~10k Tesla V100 GPUs. It seems commonly stated that the brain has the equivalent of ~...
