0
$\begingroup$

I'm reading the paper "Learning how to ask" by Qin & Eisner and in the abstract, they mention that using prompts, language models can perform tasks other than text generation. Examples include fill-in-the-blanks (BERT) and few-shots extrapolation (GPT-3).

I am not sure I understood correctly what the authors mean by few-shots extrapolation. Do they mean extracting factual and other types of knowledge by using a few examples? Thanks.

$\endgroup$
2
$\begingroup$

Here is the GPT-3 paper.

The "shot" are the number of example question/answer pairs provided to the ML model, before it is asked to answer a question by itself.

For each task, we evaluate GPT-3 under 3 conditions: (a) “few-shot learning”, or in-context learning where we allow as many demonstrations as will fit into the model’s context window (typically 10 to 100), (b) “one-shot learning”,where we allow only one demonstration, and (c) “zero-shot” learning, where no demonstrations are allowed and only an instruction in natural language is given to the model.

Figure 2.1, on page 7, provides a good example.

In the zero-shot case, the model is asked to complete the following text:

Translate English to French:
cheese =>

In the few-shot case, the model is asked to complete the following text: ("girafe" is spelled this way in the paper)

Translate English to French:
sea otter => loutre de mer
peppermint => menthe poivrée
plush girafe => girafe peluche
cheese =>

This is a 3-shot example. The text (quoted above) says the few-shot case allows 10 to 100 demonstrations; perhaps they omitted most of them to demonstrate the point more concisely.

They contrast this with "fine-tuning", where the model is explicitly trained on the task at hand. With "fine-tuning" the model is trained for a while with training data like sea otter => loutre de mer and then asked to complete cheese =>. With few-shot extrapolation, the model is not trained on these examples - the examples are only part of the text it is meant to complete.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.