Do transformers (e.g. BERT) have an unlimited input size?

Question

There are various sources on the internet that claim that BERT has a fixed input size of 512 tokens (e.g. this, this, this, this ...). This magical number also appears in the BERT paper (Devlin et al. 2019), the RoBERTa paper (Liu et al. 2019) and the SpanBERT paper (Joshi et al. 2020).

The going wisdom has always seemed to me that when NLP transitioned from recurrent models (RNN/LSTM Seq2Seq, Bahdanau ...) to transformers, we traded variable-length input for fixed-length input that required padding for shorter sequences and could not extend beyond 512 tokens (or whatever other magical number you want to assign your model).

However, come to think of it, all the parameters in a transformer (Vaswani et al. 2017) work on a token-by-token basis: the weight matrices in the attention heads and the FFNNs are applied tokenwise, and hence their parameters are independent of the input size. Am I correct that a transformer (encoder-decoder, BERT, GPT ...) can take in an arbitrary amount of tokens even with fixed parameters, i.e., the amount of parameters it needs to train is independent of the input size?

I understand that memory and/or time will become an issue for large input lengths since attention is O(n²). This is, however, a limitation of our machines and not of our models. Compare this to an LSTM, which can be run on any sequence but compresses its information into a fixed hidden state and hence blurs all information eventually. If the above claim is correct, then I wonder: What role does input length play during pre-training of a transformer, given infinite time/memory?

Intuitively, the learnt embedding matrix and weights must somehow be different if you were to train with extremely large contexts, and I wonder if this would have a positive or a negative impact. In an LSTM, it has negative impact, but a transformer doesn't have its information bottleneck.

Answer 1

You are right that a transformer can take in an arbitrary amount of tokens even with fixed parameters, excluding the positional embedding matrix, whose size directly grows with the maximum allowed input length.

Apart from memory requirements (O(n²)), the problem transformers have regarding input length is that they don't have any notion of token ordering. This is why positional encodings are used. They introduce ordering information into the model. This, however, implies that the model needs to learn to interpret such information (precomputed positional encodings) and also learn such information (trainable positional encodings). The consequence of this is that, during training, the model should see sequences that are as long as those at inference time because for precomputed positional encodings it may not correctly handle the unseen positional information and for learned positional encodings the model simply hasn't learned to represent them.

In summary, the restriction in the input length is driven by:

• Restrictions in memory: the longer the allowed input, the more memory is needed (quadratically), which doesn't play well with limited-memory devices.
• Need to train with sequences of the same length as the inference input due to the positional embeddings.

If we eliminate those two factors (i.e. infinite memory and infinite-length training data), you could set the size of the positional embeddings to an arbitrarily large number, hence allowing arbitrarily long input sequences.

Note, however, that due to the presence of the positional embeddings, there will always be a limit in the sequence length (however large it may be) that needs to be defined in advance to determine the size of the embedding matrix.