I know that in the math on which the transformer is based there is no restriction on the length of input. But I still can’t understand why we should fix it in the frameworks (PyTorch). Because of this problem Transformer-XL has been created.

Can you explain to me where this problem is hiding, please?


2 Answers 2


The restriction in the maximum length of the transformer input is due to the needed amount of memory to compute the self-attention over it.

The amount of memory needed by the self-attention in the Transformer is quadratic on the length of the input. This means that increasing the maximum length of the input, increases drastically the needed memory for self-attention. The maximum length is that which makes the model use up the whole memory of the GPU for at least one sentence (once the other elements of the model are also taken into account, like the embeddings which take a lot of memory).

Transformer-XL is certainly a way to take into account as much context as possible in language modeling (its role is analogous to truncated back-propagation through time in LSTM language models). However, the gradients are not propagated through the attention over the memory segment, only through the current segment.

There have been several architectural attempts to reduce the amount of memory needed by transformers, like using locality-constraints in the attention (Dynamic Convolutions model) or using locality-sensitive hashing (Reformer model).

There have been other implementation attempts, like gradient checkpointing(e.g. this), which is a general technique to run computations that don't fit at once in the GPU memory

  • 1
    $\begingroup$ But then how come people tend to talk about context size as if it's an actual parameter of the architecture? Like, higher batch sizes also increase memory footprint, but you wouldn't say "my architecture has an inherent max batch size of 128". You would just say "you want more batch size, you need more RAM" $\endgroup$
    – Jack M
    Commented Aug 17, 2023 at 10:47
  • $\begingroup$ @JackM maybe this answer can help with your doubt. If it's not enough, please create a new question. $\endgroup$
    – noe
    Commented Aug 17, 2023 at 15:03

Firstly, fixing the input size of a model is more of an architectural decision than a problem. By fixing input size we:

  1. Limit the GPU memory usage while training
  2. Reduce the training time per epoch
  3. Reduce evaluation time per input sample

If you want to, you can train your own transformer model by increasing the size of the input or increasing the number of attention heads or increasing the size of the hidden state.

Secondly, Transformer-XL doesn't address the problem of fixed-length input rather it addresses the problem caused due to fixed-length context size. In a Vanilla transformer, context size is fixed as a result sub-optimal output is generated sometimes.

In order to understand the problem better, let's assume you are reading a book and you can give attention to one page at a time and you need to look for an answer which is on the next page. Given your attention is only one page can you relate to a question on the current page whose answer is on the next page?

The answer is no. So to solve the problem you need to increase attention size or make it so you can go through more pages to look for the answer.

In Transformer-XL[1], context size is made dynamic by adding a notion of recurrence on attention heads and so the model can predict more accurately.

I hope this clears your doubt. For more details, you can go through the introduction section of Transformer-XL[1] paper.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.