In a transformer, the target sequence shifted right is placed in the input of the transformer decoder, and a upper triangular mask is provided so that the attention layer will not depend "tokens from the future". The output of the encoder stage is not relevant for this question and it is assumed to be fixed.

The output of the transformer is a tensor of shape (sequence_lenght, dictionary_lenght) that contains the probabilities of each possible token, for each element of the sequence.

Are the probabilities predicted for the Nth element of the output sequence always the same for any two inputs sequences that have the same tokens for token 0 to token N of the sequences?

More formally: Consider two target sequences S1 = [<begin>, x0, ..., xN] and S2 = [<begin>, X0, ..., xM] where N < M, that is: two sequences that contains the exact same tokens up to token with index N, and then the second contains some more tokens. Assume decoder: Tensor(sequence_length) -> Tensor(sequence_length, dictionary_length) is a function that implements a decoder that has already been bound to a mask and to a encoder output.

Is it true that decoder(S1)[X][Y] == decoder(S2)[X][Y] for all X in [0..N], Y in [0..dictionary_lenght]?

My understanding is that this is the entire point of a transformer, when the loss is backpropagated the transformer will learn as if it was only aware of the tokens it generated up to that point, but this is beyond my ability to prove, and i have not seen it being explicitly said anywhere.


1 Answer 1


Yes, you are right, Transformer decoders are meant to only attend to previous tokens. From the original article:

[...] We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position i can depend only on the known outputs at positions less than i.

And yes, Transformers are deterministic, in the sense that for a given prefix, you will always get the same next token probabilities. This is because they only attend to previous tokens and there is no source of stochasticity whatsoever in the Transformer itself. Note that the decoding process can be stochastic, but that is another matter.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.