How are Learned Latent Arrays for the Perceiver Resampler in DeepMind's Flamingo Vision-Language Model Actually Calculated? By which Technique?

Question

In "Flamingo: a Visual Language Model for Few-Shot Learning" (Alayrac et al. 2022) https://arxiv.org/abs/2204.14198 DeepMind makes use of "learned latent queries" in their "Perceiver Resampler" to ensure that parameters do not scale quadratically the way they do with Transformers. The authors cite the DeepMind article "Perceiver: General Perception with Iterative Attention" (Jaegle et al., 2021) https://arxiv.org/abs/2103.03206 as inspiration for their creation of Perceiver Resamplers. Perceivers from the Jaegle et al. article involve "learned latent queries" (i.e., queries from learned latent arrays) that cross-attend to image feature-based keys and values. My understanding is that these learned latent arrays are a reduced dimensional representation of the visual feature arrays that are the outputs of the Vision Encoder. However, the Flamingo paper does not explain how the learned latent array is actually computed from the original visual feature array from the Vision Encoder.

In terms of the Perceiver from Jaegle et al., the authors seem to hint that learned latent arrays may be created through some kind of clustering algorithm. They state, "The model can also be seen as performing a fully end-to-end clustering of the inputs with latent positions as cluster centres, leveraging a highly asymmetric cross-attention layer" (pg. 3). But if they use a clustering algorithm of some kind to produce the learned latent arrays, as far as I can see they do not explain how exactly such an algorithm could be reproduced for use in code, and so they leave it somewhat to the imagination to figure out.

I have 2 questions:

1) Are these learned latent arrays learned from the visual features 𝑋𝑓 that come from Flamingo's Visual Encoder? If not, where are they being learned from?

2) If so, how exactly (in a way that I might try to replicate the process) are learned latent arrays calculated from visual feature arrays that are outputs from the Vision Encoder?

Thank you for your help.

Source for image below (Alayrac et al. 2022, pg. 11):

Answer 1

You can think of attention, as used in Transformers or Perceiver as a soft differentiable database. The keys, values and queries are continuously valued vectors and instead of computing a perfect result, i.e. a single query match, you will get weightings of the values depending on the vector similarity of the queries to the keys.

Now for your question, suppose you have some feature vectors from a vision encoder that serve as keys and values. In a sense, the table of our the database is already filled with data, we just have to figure out how to query it. If we want to solve the problem of image classification for instance, and the label of an image is 'dog', there might be some visual tokens that represent this, say, in the form of a texture feature. The latent vectors in this case act as queries which extract information from the input data and need to be aligned in such a way, that they extract the necessary information to perform the prediction task.

Since the whole model is differentiable, we can backpropagate the loss and shift the latent vectors in a direction that extracts more meaningful information from the inputs.

In Jaegle et al. they refer to the latent vectors as clusters because each of them might in fact something like a specific latent feature which can be seen as an implicit clustering of the data within the internal representations.

The latent vectors are, however, not pre-computed from any features but initialized randomly at the beginning and then learned via gradient descent as described above.