# Do transformers output probabilities depend only on previous tokens?

In a transformer, the target sequence shifted right is placed in the input of the transformer decoder, and a upper triangular mask is provided so that the attention layer will not depend "tokens from the future". The output of the encoder stage is not relevant for this question and it is assumed to be fixed.

The output of the transformer is a tensor of shape (sequence_lenght, dictionary_lenght) that contains the probabilities of each possible token, for each element of the sequence.

Are the probabilities predicted for the Nth element of the output sequence always the same for any two inputs sequences that have the same tokens for token 0 to token N of the sequences?

More formally: Consider two target sequences S1 = [<begin>, x0, ..., xN] and S2 = [<begin>, X0, ..., xM] where N < M, that is: two sequences that contains the exact same tokens up to token with index N, and then the second contains some more tokens. Assume decoder: Tensor(sequence_length) -> Tensor(sequence_length, dictionary_length) is a function that implements a decoder that has already been bound to a mask and to a encoder output.

Is it true that decoder(S1)[X][Y] == decoder(S2)[X][Y] for all X in [0..N], Y in [0..dictionary_lenght]?

My understanding is that this is the entire point of a transformer, when the loss is backpropagated the transformer will learn as if it was only aware of the tokens it generated up to that point, but this is beyond my ability to prove, and i have not seen it being explicitly said anywhere.