There are various sources on the internet that claim that BERT has a fixed input size of 512 tokens (e.g. this, this, this, this ...). This magical number also appears in the BERT paper (Devlin et al. 2019), the RoBERTa paper (Liu et al. 2019) and the SpanBERT paper (Joshi et al. 2020).
The going wisdom has always seemed to me that when NLP transitioned from recurrent models (RNN/LSTM Seq2Seq, Bahdanau ...) to transformers, we traded variable-length input for fixed-length input that required padding for shorter sequences and could not extend beyond 512 tokens (or whatever other magical number you want to assign your model).
However, come to think of it, all the parameters in a transformer (Vaswani et al. 2017) work on a token-by-token basis: the weight matrices in the attention heads and the FFNNs are applied tokenwise, and hence their parameters are independent of the input size. Am I correct that a transformer (encoder-decoder, BERT, GPT ...) can take in an arbitrary amount of tokens even with fixed parameters, i.e., the amount of parameters it needs to train is independent of the input size?
I understand that memory and/or time will become an issue for large input lengths since attention is O(n²). This is, however, a limitation of our machines and not of our models. Compare this to an LSTM, which can be run on any sequence but compresses its information into a fixed hidden state and hence blurs all information eventually. If the above claim is correct, then I wonder: What role does input length play during pre-training of a transformer, given infinite time/memory?
Intuitively, the learnt embedding matrix and weights must somehow be different if you were to train with extremely large contexts, and I wonder if this would have a positive or a negative impact. In an LSTM, it has negative impact, but a transformer doesn't have its information bottleneck.