Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I know that in the math on which the transformer is based there is no restriction on the length of input. But I still can’t understand why we should fix it in the frameworks (PyTorch). Because of this problem Transformer-XL has been created.
Can you explain to me where this problem is hiding, please?
The restriction in the maximum length of the transformer input is due to the needed amount of memory to compute the self-attention over it.
The amount of memory needed by the self-attention in the Transformer is quadratic on the length of the input. This means that increasing the maximum length of the input, increases drastically the needed memory for self-attention. The maximum length is that which makes the model use up the whole memory of the GPU for at least one sentence (once the other elements of the model are also taken into account, like the embeddings which take a lot of memory).
Transformer-XL is certainly a way to take into account as much context as possible in language modeling (its role is analogous to truncated back-propagation through time in LSTM language models). However, the gradients are not propagated through the attention over the memory segment, only through the current segment.
There have been several architectural attempts to reduce the amount of memory needed by transformers, like using locality-constraints in the attention (Dynamic Convolutions model) or using locality-sensitive hashing (Reformer model).
There have been other implementation attempts, like gradient checkpointing(e.g. this), which is a general technique to run computations that don't fit at once in the GPU memory
Firstly, fixing the input size of a model is more of an architectural decision than a problem. By fixing input size we:
If you want to, you can train your own transformer model by increasing the size of the input or increasing the number of attention heads or increasing the size of the hidden state.
Secondly, Transformer-XL doesn't address the problem of fixed-length input rather it addresses the problem caused due to fixed-length context size. In a Vanilla transformer, context size is fixed as a result sub-optimal output is generated sometimes.
In order to understand the problem better, let's assume you are reading a book and you can give attention to one page at a time and you need to look for an answer which is on the next page. Given your attention is only one page can you relate to a question on the current page whose answer is on the next page?
The answer is no. So to solve the problem you need to increase attention size or make it so you can go through more pages to look for the answer.
In Transformer-XL[1], context size is made dynamic by adding a notion of recurrence on attention heads and so the model can predict more accurately.
I hope this clears your doubt. For more details, you can go through the introduction section of Transformer-XL[1] paper.
