I am a bit perplex from the transformer-XL architecture that is claimed to solve the issue of context fragmantation. I probably understood it wrong but it looks like all the transformer-XL is doing, is being 2 times larger than the normal one, and that you translate at each iterations. Plus a fancy new way of doing positional encoding.

Am I interpreting it wrong ?


I think your interpretation does not focus on the right places.

First, let's understand where the predecessor idea of Transformer XL in LSTM Language Models: Truncated Back-Propagation Through Time (TBPTT). In LSTM Language Models, the training data text is concatenated into a single very long sequence. From this sequence, we create training batches in a way that for batch $k$, the text sequence at position $i$ is the continuation of the sequence at the same position in batch $k-1$. When training the LSTM, the last hidden states generated at batch $k-1$ were used as initial hidden states when processing batch $k$. The gradients were not propagated through batches, only the values of the hidden states. This allowed the LSTM to be able to profit from unlimited-length context.

The same idea cannot be directly applied to Transformers, as there is not a single hidden state, but as many hidden states as the number of tokens in the sequence times the number of layers.

TransformerXL solves that by incorporating all those hidden states from the previous batch into the computation of the new batch, without propagating the gradients (see figure below from the original article).

enter image description here

This, however, poses a problem in how the positional information in managed in the Transformer, as the positional embeddings encode absolute positions. Therefore, the authors reformulated them to be relative.

Therefore, the main advantage of Transformer XL over vanilla Transformer LMs is that it can profit from unlimited-length context.

Note that Transformer XL is "only" a Language Model, it does not translate, just computes probabilities of the next token based on the previous ones.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.