How can models like Mosaic's MPT-7b or Bloombergs BLOOMGPT take in so many tokens?

Question

I've read the paper on ALiBi, and I understand that these models are biasing the values made in the query/key multiplication.

But from my understanding, when I build the actual model I give it N input nodes. When I train a model I give it vectors of length N. How then at inference can I give it vectors of length greater than N? Am I misunderstanding how the multiplication of key and query works? Can there be keys of any length?

Edit: I guess my question includes, why isn't there a multiplication error when I use longer keys in my inference?

Answer 1

In the attention blocks, Keys and Values are of the same length, but Queries are not necessarily of the same length as the former. For instance, in the encoder-decoder attention blocks, the Keys and Values come from the encoder representations while the Queries come from the decoder representations (which are computed from a completely different sequence than the encoder ones).

In the Transformer model, the only factor limiting the length of the inputs is the positional embeddings (see this answer for details. In the ALiBi paper, they replace them with a non-trainable approach and, experimentally, the approach is shown to extrapolate to longer sequences than those seen during training.