Firstly, fixing the input size of a model is more of an architectural decision than a problem. By fixing input size we:
- Limit the GPU memory usage while training
- Reduce the training time per epoch
- Reduce evaluation time per input sample
If you want to, you can train your own transformer model by increasing the size of the input or increasing the number of attention heads or increasing the size of the hidden state.
Secondly, Transformer-XL doesn't address the problem of fixed-length input rather it addresses the problem caused due to fixed-length context size. In a Vanilla transformer, context size is fixed as a result sub-optimal output is generated sometimes.
In order to understand the problem better, let's assume you are reading a book and you can give attention to one page at a time and you need to look for an answer which is on the next page. Given your attention is only one page can you relate to a question on the current page whose answer is on the next page?
The answer is no. So to solve the problem you need to increase attention size or make it so you can go through more pages to look for the answer.
In Transformer-XL, context size is made dynamic by adding a notion of recurrence on attention heads and so the model can predict more accurately.
I hope this clears your doubt. For more details, you can go through the introduction section of Transformer-XL paper.