3
$\begingroup$

What exactly are the parameters in GPT-3's 175 billion parameters? Are these the words in text on which model is trained?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

The parameters in GPT-3, like any neural network, are the weights and biases of the layers.

From the following table taken from the GTP-3 paper

GPT-3 Table

there are different versions of GPT-3 of various sizes. The more layers a version has the more parameters it has since it has more weights and biases. Regardless of the model version, the words it was trained on are the 300 billion tokens the caption references with what appears to be around 45TB of data scraped from the internet.

Improve this answer
$\endgroup$
1
$\begingroup$

This has been answered in ai.stackexchange.com:

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are hyper-parameters which basically means they are user specified, whereas weights are what the learning algorithm will learn through training.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So basically the stuff the model learns? $\endgroup$
    – orome
    Feb 20 at 18:41
  • 1
    $\begingroup$ Yes, that's it. $\endgroup$
    – noe
    Feb 20 at 18:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.