If the masking were only applied in the first layer, the self-attention would lead the should-be-masked positions to have non-zero values at each subsequent layer, and therefore influence the values at the other positions.

While the training may compensate for this, we should take into account that the number of should-be-masked positions is variable even for the same sentence, as the sequence length (and therefore the amount of padding) of each batch depends on the longest sequence in it, hence making it difficult for the padding to compensate for this.

If the masking were only applied in the first layer, the self-attention in the subsequent layers would bring to each position information from future tokens.

Let's break it down with numbers:

• At layer $i$, if causal masking is applied, the output at position $t$ contains information about layer $i-1$ at positions $1..t-1$, that is, $L_{i,t} = f_i(L_{i-1,1},...,L_{i-1,t-1})$.

• If no causal masking is applied, then the output at position $t$ contains information about layer $i-1$ at all positions in the sequence of length $T$, that is, positions $1..T$ $L_{i,t} = f_i(L_{i-1,1},...,L_{i-1,T})$

• If causal masking is applied at layer 1 (the first layer) but not at layer 2 or 3, we obtain that for position t at layer 3 we would have: $L_{3,t} = f_3(L_{2,1},...,L_{2,T}) = f_3(L_{2,1},...,f_1(L_{1,1},...,L_{1,T}))$, which means that position $t$ contains information from future tokens, as $T > t$.

Note: The original answer was wrong and was completely edited. To check the original answer, refer to the post timeline.