1
$\begingroup$

GPT-3 has 175 billion parameters, required ~$3.114 * 10^{23}$ FLOPS, and took approximately one month to train on ~10k Tesla V100 GPUs. It seems commonly stated that the brain has the equivalent of ~100 trillion parameters. I was wondering what kind of compute would be required for training a transformer of this size. Would it simply be ~$10^3$ times more FLOPs?

In general, how does compute required scale with respect to model parameters for transformers, neural networks, CNNs, and other popular deep learning models?

Improve this question
$\endgroup$
1
  • $\begingroup$ ~100 trillion parameters for the whole control over your body... you should only take in account the part of the brain required for the task you are learning, (eg for speech, Broca's area, which has much less neurons than the whole brain) $\endgroup$
    – anon
    Commented Jun 28, 2022 at 10:15

1 Answer 1

Reset to default
1
$\begingroup$

Recent models using transformers like Lambda have less than 100 trillion parameters, and it answers much better than most humans (if not all, as it has a huge volume of knowledge).

I mean that the human brain and the artificial brain are not comparable, and it doesn't mean that having 100 trillion parameters in an artificial brain would be equivalent to a human one.

Then, the correlation between parameters and FLOPs is quite linear indeed. I have created a table in an article to have a rough estimation order of magnitude because very little information exists on this topic.

enter image description here

Source: https://medium.com/p/1cd2225fd0f2

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.