0
$\begingroup$

In the documentation for GPT-3 API, it says One limitation to keep in mind is that, for most models, a single API request can only process up to 2,048 tokens (roughly 1,500 words) between your prompt and completion.

In the documentation for fine tuning model, it says The more training samples you have, the better. We recommend having at least a couple hundred examples. in general, we've found that each doubling of the dataset size leads to a linear increase in model quality.

My question is, does the 1,500 words limit also apply to fine tune model? Does "Doubling of the dataset size" mean number of training datasets instead of size of each training dataset?

$\endgroup$

1 Answer 1

1
$\begingroup$

A sample/example refers to an individual sentence or piece of text.

The training dataset is the list of examples you have.

Doubling the data size means doubling the number of examples.

The 2048 token limit applies to each of the examples used to fine-tune the model. This is the maximum sequence length that GPT-3 can process (see this Twitter thread).

Note that the equivalence of 2048 tokens to 1500 words is an estimation for English. For other languages, especially those with different scripts (e.g. Chinese), the number of tokens needed to represent one word/character may be much higher.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.