0
$\begingroup$

After exhaustively reading the many blogs and papers on Transformers-XL, I still have some questions before I can say that I understand Transformer-XL (and by extension XLNet). Any help in this regard is hugely appreciated.

  1. when we say hidden states are transferred from one segment to another, what exactly is included in these hidden states? Are the weights of the networks implementing the attention mechanism (i.e. calculating the Q, K and V) included? Are the weights involved in calculating the input word embedding included in the hidden state?
  2. When the hidden states are transferred during recurrence, is this transfer from the encoder of one segment to the encoder of the next segment? Or is it from the decoder of the current segment to the encoder of the next segment? Is the decoder involved at all in the hidden state transfer?
  3. I see images like the following in the following in the papers and blogs. what do the dots represent? encoders? decoders? or an entire unit? I guess the answer to my second question will shed a light on this one too. From Transformer-XL Google page

Thank you

$\endgroup$
0
$\begingroup$

By hidden states, they mean outputs of the layers, i.e., what you get after the feed-forward sub-layer. For Transformer-XL, it is important that these are also what you use as an input to the self-attention. Therefore, at inference time, if you want to compute the states recursively by segments (presumably because you cannot fit the entire input int he memory), this is the only thing you need to remember from the previous steps to continue the computation.

There is no encoder, you can imagine Transformer-XL as a decoder-only model. Transfering the states just means remembering them, so you can do the self-attention over them, but you can no longer back-propagate through them because you only remember the values and the entire graph telling how you got them.

The dots in the scheme correspond to the hidden states: one state per input subword and per layer. The lines between them are the self-attention links.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks @Jindřich. So I guess your answer applies to the XLNet architecture as well? $\endgroup$ – Vistas Aug 20 at 13:47
  • $\begingroup$ Yes, but the masking scheme in XL-Net is more complicated. Instead of doing left-to-right generation, they sample from a set all order permutations. $\endgroup$ – Jindřich Aug 20 at 14:12
  • $\begingroup$ That part I got :) Thanks for the responses. $\endgroup$ – Vistas Aug 20 at 14:30
0
$\begingroup$

Based on what @Jindřich replied above, I made another literature survey. I found an architecture schema for Transformer-XL from the paper Language Modelling for Source Code Using Transformer-XL which I copy here to serve as a reference/guide for those with the same question in future:

enter image description here

Note that the paper doesn't appear to be peer-reviewed yet. So corrections (if any) suggested by the readers are most welcome.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.