One type of statement about zero-shot and few-shot learning in the literature I continually come across is that these models can predict new unseen classes at inference time for which they were never trained on. However, such sources typically do not explain exactly what they mean.
Meta-learning/in-context learning-based zero-shot/few-shot learning models like Flamingo and CLIP rely on 1) a pre-training stage where a massive base vision-language model has been trained on millions to billions of images and text examples, and 2) an inference stage where a prompt with anywhere from 0 to a just a few examples are presented to the model inside a prompt's "support set", along with an image or image + question "query" (see the diagram from the Flamingo paper below) which asks the model to generate an answer to the query.
Diagram below is the Flamingo model paper (Alayrac et al., pg. 16):
My Questions
As a result, it is unclear to me whether scholars' statements about zero-/few-shot learning models being able to predict "unseen" classes at inference time refer to the model never having been pre-trained on these unseen classes in the base model, whether they mean the model has never seen the unseen examples in the support set at inference time, or whether they mean both. Does anyone know?
Can someone explain exactly how the Flamingo model by Alayrac et al., 2022, or the CLIP model by Radford et al., 2021 (both of which are pre-trained using contrastive loss) would be able to predict a class at inference time which has never before been seen by the model? How would this even be possible if the model does not know the label of an unseen image?
