After exhaustively reading the many blogs and papers on Transformers-XL, I still have some questions before I can say that I understand Transformer-XL (and by extension XLNet). Any help in this regard is hugely appreciated.
- when we say hidden states are transferred from one segment to another, what exactly is included in these hidden states? Are the weights of the networks implementing the attention mechanism (i.e. calculating the Q, K and V) included? Are the weights involved in calculating the input word embedding included in the hidden state?
- When the hidden states are transferred during recurrence, is this transfer from the encoder of one segment to the encoder of the next segment? Or is it from the decoder of the current segment to the encoder of the next segment? Is the decoder involved at all in the hidden state transfer?
- I see images like the following in the following in the papers and blogs. what do the dots represent? encoders? decoders? or an entire unit? I guess the answer to my second question will shed a light on this one too.