There are various sources on the internet that claim that BERT has a fixed input size of 512 tokens (e.g. this, this, this, this ...). This magical number also appears in the BERT paper (Devlin et al. 2019), the RoBERTa paper (Liu et al. 2019) and the SpanBERT paper (Joshi et al. 2020).

The going wisdom has always seemed to me that when NLP transitioned from recurrent models (RNN/LSTM Seq2Seq, Bahdanau ...) to transformers, we traded variable-length input for fixed-length input that required padding for shorter sequences and could not extend beyond 512 tokens (or whatever other magical number you want to assign your model).

However, come to think of it, all the parameters in a transformer (Vaswani et al. 2017) work on a token-by-token basis: the weight matrices in the attention heads and the FFNNs are applied tokenwise, and hence their parameters are independent of the input size. Am I correct that a transformer (encoder-decoder, BERT, GPT ...) can take in an arbitrary amount of tokens even with fixed parameters, i.e., the amount of parameters it needs to train is independent of the input size?

I understand that memory and/or time will become an issue for large input lengths since attention is O(n²). This is, however, a limitation of our machines and not of our models. Compare this to an LSTM, which can be run on any sequence but compresses its information into a fixed hidden state and hence blurs all information eventually. If the above claim is correct, then I wonder: What role does input length play during pre-training of a transformer, given infinite time/memory?

Intuitively, the learnt embedding matrix and weights must somehow be different if you were to train with extremely large contexts, and I wonder if this would have a positive or a negative impact. In an LSTM, it has negative impact, but a transformer doesn't have its information bottleneck.


1 Answer 1


You are right that a transformer can take in an arbitrary amount of tokens even with fixed parameters, excluding the positional embedding matrix, whose size directly grows with the maximum allowed input length.

Apart from memory requirements (O(n²)), the problem transformers have regarding input length is that they don't have any notion of token ordering. This is why positional encodings are used. They introduce ordering information into the model. This, however, implies that the model needs to learn to interpret such information (precomputed positional encodings) and also learn such information (trainable positional encodings). The consequence of this is that, during training, the model should see sequences that are as long as those at inference time because for precomputed positional encodings it may not correctly handle the unseen positional information and for learned positional encodings the model simply hasn't learned to represent them.

In summary, the restriction in the input length is driven by:

  • Restrictions in memory: the longer the allowed input, the more memory is needed (quadratically), which doesn't play well with limited-memory devices.
  • Need to train with sequences of the same length as the inference input due to the positional embeddings.

If we eliminate those two factors (i.e. infinite memory and infinite-length training data), you could set the size of the positional embeddings to an arbitrarily large number, hence allowing arbitrarily long input sequences.

Note, however, that due to the presence of the positional embeddings, there will always be a limit in the sequence length (however large it may be) that needs to be defined in advance to determine the size of the embedding matrix.

  • $\begingroup$ "amount of parameters it needs to train is independent of the input size" I think it is true for Bert, but not true for GPT-2. From the Hugging Face's implementation it looks that the parameter matrix for "learned absolute positional embeddings" scales as input window size x embedding dimension, so it grows with input window size. $\endgroup$
    – Valentas
    Mar 31, 2023 at 10:21
  • $\begingroup$ It's true for Transformers in general. Can you point to the specific line that makes you think that it depends on input size? In the linked implementation, I see that the positional embedding size (maximum representable position) is taken from config.max_position_embeddings. $\endgroup$
    – noe
    Mar 31, 2023 at 10:31
  • $\begingroup$ Yes. And that parameter seems to be initialized to be the same as n_positions, (=1024 in GPT-2). This makes sense to me and corresponds to the description in the GPT paper. $\endgroup$
    – Valentas
    Mar 31, 2023 at 11:04
  • 1
    $\begingroup$ I see the misunderstanding. The maximum number of positions needs to set to some number because of what I explained in my answer: memory restrictions and need to see during training sequences as long as during inference. Apart from that, nothing keeps you from using an arbitrarily large number from n_positions. $\endgroup$
    – noe
    Mar 31, 2023 at 11:08
  • 1
    $\begingroup$ I added a paragraph to the answer to clarify it. $\endgroup$
    – noe
    Mar 31, 2023 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.