working thru the annotated transformer, I see that every layer in both the encoder (mask paddings) and decoder (mask padding + future positions) get masked. Why couldn't it be simplified to just one mask at the first layers of encoder and decoder?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
If the masking were only applied in the first layer, the self-attention in the subsequent layers would bring to each position information from future tokens.
Let's break it down with numbers:
At layer $i$, if causal masking is applied, the output at position $t$ contains information about layer $i-1$ at positions $1..t-1$, that is, $L_{i,t} = f_i(L_{i-1,1},...,L_{i-1,t-1})$.
If no causal masking is applied, then the output at position $t$ contains information about layer $i-1$ at all positions in the sequence of length $T$, that is, positions $1..T$ $L_{i,t} = f_i(L_{i-1,1},...,L_{i-1,T})$
If causal masking is applied at layer 1 (the first layer) but not at layer 2 or 3, we obtain that for position t at layer 3 we would have: $L_{3,t} = f_3(L_{2,1},...,L_{2,T}) = f_3(L_{2,1},...,f_1(L_{1,1},...,L_{1,T}))$, which means that position $t$ contains information from future tokens, as $T > t$.
Note: The original answer was wrong and was completely edited. To check the original answer, refer to the post timeline.
-
$\begingroup$ thanks, I think I got you - let me rephrase and please approve: the output of each layer, $out(pos)$ is a function of all other positions in the sentence, as that's what self-attention does. By masking properly for each position, $out(pos)$ is a function of only the previous positions it's allowed to see. We need to do that for every layer, as in every layer looking at subsequent positions is using data you are not allowed to see. correct? $\endgroup$– ihadannyCommented Jan 18, 2021 at 18:54
-
-
$\begingroup$ and please also clarify, what do you mean by
While the training may compensate for this? how? this is data leakage and there's no way to compensate for that as it's cheating which won't be possible when testing... $\endgroup$– ihadannyCommented Jan 18, 2021 at 18:55 -
$\begingroup$ Wait, now I see that I misunderstood your first comment, sorry for reading too fast and not caching that "data you are not allowed to see" referred to future tokens. To prevent the decoder to see future tokens, it is enough to mask in the first layer. However, if we only mask the first layer, the masked positions (future tokens and padding) of subsequent layers would still be taken into account, and that is what masking at each layer aims at preventing. $\endgroup$– noeCommented Jan 18, 2021 at 19:19
-
$\begingroup$ The training may be able to compensate for the biases being added from the padding and future tokens if they were always the same (i.e. if the sequence length were the same), but the variable-length batches prevent that. $\endgroup$– noeCommented Jan 18, 2021 at 19:21