What is few-shots extrapolation?

Question

I'm reading the paper "Learning how to ask" by Qin & Eisner and in the abstract, they mention that using prompts, language models can perform tasks other than text generation. Examples include fill-in-the-blanks (BERT) and few-shots extrapolation (GPT-3).

I am not sure I understood correctly what the authors mean by few-shots extrapolation. Do they mean extracting factual and other types of knowledge by using a few examples? Thanks.

Answer 1

Here is the GPT-3 paper.

The "shot" are the number of example question/answer pairs provided to the ML model, before it is asked to answer a question by itself.

For each task, we evaluate GPT-3 under 3 conditions: (a) “few-shot learning”, or in-context learning where we allow as many demonstrations as will fit into the model’s context window (typically 10 to 100), (b) “one-shot learning”,where we allow only one demonstration, and (c) “zero-shot” learning, where no demonstrations are allowed and only an instruction in natural language is given to the model.

Figure 2.1, on page 7, provides a good example.

In the zero-shot case, the model is asked to complete the following text:

Translate English to French:
cheese =>

In the few-shot case, the model is asked to complete the following text: ("girafe" is spelled this way in the paper)

Translate English to French:
sea otter => loutre de mer
peppermint => menthe poivrée
plush girafe => girafe peluche
cheese =>

This is a 3-shot example. The text (quoted above) says the few-shot case allows 10 to 100 demonstrations; perhaps they omitted most of them to demonstrate the point more concisely.

They contrast this with "fine-tuning", where the model is explicitly trained on the task at hand. With "fine-tuning" the model is trained for a while with training data like sea otter => loutre de mer and then asked to complete cheese =>. With few-shot extrapolation, the model is not trained on these examples - the examples are only part of the text it is meant to complete.