Recent models like the GPT-3 Language Model (Brown et al., 2020) and the Flamingo Visual-Language Model (Alayrac et al., 2022) use in-context few-shot learning. The models are able to make highly accurate predictions even when only presented with a "few" support examples. See diagram below (from Brown et al., 2020).
Yet, it is unclear to me how these models theoretically work behind the scenes, and why they perform so well. The explanation appears to be that few-shot learning works because the model looks at the task description, then looks at the support examples (which are successful examples of how the given task can be fulfilled), and then based on the model's understanding of what the assigned task is and its understanding of the examples given of how the task could be successfully fulfilled it is then able to understand what it is supposed to predict based on the prompt.
Generally speaking, the more support examples the model sees at inference time, the better it will perform (but there is a point at which continuing to add further support examples does not increase performance). However, given that traditional machine learning models need to train on thousands of examples, it would seem unlikely that a model could really fulfill a task just based on a few examples.
I understand that these models are built on huge pre-trained Language Models or Vision-Language Models having billions of parameters. But is there a commonly understood explanation of how these models are actually able to work (e.g., mathematical intuition) beyond what I have described?
Since these specific models (GPT-3 and Flamingo use "in-context learning," which I understand to be the same as "meta-learning," is it the case that what is actually happening in these models is that the massive pre-trained language and/or vision models they are built on are able to learn many tasks, and that consequently at inference time the model is able to learn from the few-shot prompt it is given what the new task being asked of it is, and also is able to learn the image/text query presented to it at inference time because it has been pre-trained on massive amounts of examples it can refer back to?
And is there a commonly accepted explanation of why these models actually work so well? Or are these three questions still a matter of debate among ML scholars?