If I understand correctly, the Key
hidden layer in the Transformer XL is of size 2L * d
, where L
is the segment length and d
is the embedding dimension.
concatenation of two hidden sequences along the length dimension
Therefore, the size of the attention matrix would be L X 2L
, where row i
represents the attention Query i
should apply to each of the 2L Keys
.
That is, the self attention window length = 2 X segment length.
However, in the following image from the paper, the segment length is 4 and there are only 4 lines linked to each node. Shouldn't there be 4 * 2 = 8 lines from each node?