In "Flamingo: a Visual Language Model for Few-Shot Learning" (Alayrac et al. 2022) https://arxiv.org/abs/2204.14198 DeepMind makes use of "learned latent queries" in their "Perceiver Resampler" to ensure that parameters do not scale quadratically the way they do with Transformers. The authors cite the DeepMind article "Perceiver: General Perception with Iterative Attention" (Jaegle et al., 2021) https://arxiv.org/abs/2103.03206 as inspiration for their creation of Perceiver Resamplers. Perceivers from the Jaegle et al. article involve "learned latent queries" (i.e., queries from learned latent arrays) that cross-attend to image feature-based keys and values. My understanding is that these learned latent arrays are a reduced dimensional representation of the visual feature arrays that are the outputs of the Vision Encoder. However, the Flamingo paper does not explain how the learned latent array is actually computed from the original visual feature array from the Vision Encoder.
In terms of the Perceiver from Jaegle et al., the authors seem to hint that learned latent arrays may be created through some kind of clustering algorithm. They state, "The model can also be seen as performing a fully end-to-end clustering of the inputs with latent positions as cluster centres, leveraging a highly asymmetric cross-attention layer" (pg. 3). But if they use a clustering algorithm of some kind to produce the learned latent arrays, as far as I can see they do not explain how exactly such an algorithm could be reproduced for use in code, and so they leave it somewhat to the imagination to figure out.
I have 2 questions:
1) Are these learned latent arrays learned from the visual features 𝑋𝑓 that come from Flamingo's Visual Encoder? If not, where are they being learned from?
2) If so, how exactly (in a way that I might try to replicate the process) are learned latent arrays calculated from visual feature arrays that are outputs from the Vision Encoder?
Thank you for your help.