7
$\begingroup$

Recent models like the GPT-3 Language Model (Brown et al., 2020) and the Flamingo Visual-Language Model (Alayrac et al., 2022) use in-context few-shot learning. The models are able to make highly accurate predictions even when only presented with a "few" support examples. See diagram below (from Brown et al., 2020).

enter image description here

Yet, it is unclear to me how these models theoretically work behind the scenes, and why they perform so well. The explanation appears to be that few-shot learning works because the model looks at the task description, then looks at the support examples (which are successful examples of how the given task can be fulfilled), and then based on the model's understanding of what the assigned task is and its understanding of the examples given of how the task could be successfully fulfilled it is then able to understand what it is supposed to predict based on the prompt.

Generally speaking, the more support examples the model sees at inference time, the better it will perform (but there is a point at which continuing to add further support examples does not increase performance). However, given that traditional machine learning models need to train on thousands of examples, it would seem unlikely that a model could really fulfill a task just based on a few examples.

My Questions:

  • I understand that these models are built on huge pre-trained Language Models or Vision-Language Models having billions of parameters. But is there a commonly understood explanation of how these models are actually able to work (e.g., mathematical intuition) beyond what I have described?

  • Since these specific models (GPT-3 and Flamingo use "in-context learning," which I understand to be the same as "meta-learning," is it the case that what is actually happening in these models is that the massive pre-trained language and/or vision models they are built on are able to learn many tasks, and that consequently at inference time the model is able to learn from the few-shot prompt it is given what the new task being asked of it is, and also is able to learn the image/text query presented to it at inference time because it has been pre-trained on massive amounts of examples it can refer back to?

  • And is there a commonly accepted explanation of why these models actually work so well? Or are these three questions still a matter of debate among ML scholars?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I like the question. An answer is going to be quite a bit of work, though! I've never thought of GPT3, etc. as meta learning; it might be helpful if you described/linked to what you mean by meta learning? $\endgroup$
    – Darren Cook
    Commented Oct 31, 2022 at 19:12
  • $\begingroup$ I should clarify that the GPT3 authors see a slight distinction between the terms, although the processes go hand-in-hand (and I think may be the same). They show an ambiguous diagram on pg. 3 of pre-training with learning via SGD (called the "outer loop"), and an "inner loop" process of task learning referred to as "in-context learning", whereas the inner-loop + outer loop = "meta-learning". On pg. 4, it says, "These terms are intended to remain agnostic on the question of whether the model learns new tasks from scratch at inference time or simply recognizes patterns seen during training..." $\endgroup$
    – user141493
    Commented Nov 1, 2022 at 6:20
  • $\begingroup$ They state, "meta-learning – which in the context of language models means the model develops a broad set of skills and pattern recognition abilities at training time, and then uses those abilities at inference time to rapidly adapt to or recognize the desired task. Recent work attempts to do this via what we call “in-context learning”, using the text input of a pretrained language model as a form of task specification: the model is conditioned on a natural language instruction and/or a few demonstrations of the task and is then expected to complete further instances of the task...” (pg. 4). $\endgroup$
    – user141493
    Commented Nov 1, 2022 at 7:13
  • 2
    $\begingroup$ This paper from Stanford AI was helpful to me in understanding this idea better: ai.stanford.edu/blog/understanding-incontext $\endgroup$
    – John Lam
    Commented Jan 6, 2023 at 18:03

1 Answer 1

Reset to default
3
$\begingroup$

I highly recommend you read Microsoft's recent paper about In Context Learning. Although the focus is on LLM I think it can be generalised to other models.

The idea is to consider models as mesa|meta-optimisers (optimisers at inference time).

They approximately show that the model performs implicit gradient descent (and thus implicit fine-tuning) at inference time. Obviously, the gradient descent doesn't modify the models' weights, but it modifies the attention mechanism (as would fine-tuning by modifying the weights).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.