I'm working through Attention is All you Need, and I have a question about masking in the decoder. It's stated that masking is used to ensure the model doesn't attend to any tokens in the future (not yet predicted), so it can be used autoregressively during inference.

I don't understand how masking is used during inference. When the encoder is given an unseen sample with no ground truth output or prediction, it seems to me that there is nothing to mask, since there aren't any output tokens beyond what the decoder has already produced. Is my understanding of masking correct?


  • $\begingroup$ Welcome to the forum! Your question is not clear. Please add some context. $\endgroup$
    – Peter
    Aug 26 '20 at 18:01
  • $\begingroup$ Okay, I tried to clarify it a bit. If it still is unclear, are there any clarifying questions you could ask? $\endgroup$
    – David Rein
    Aug 26 '20 at 20:26

The trick is that you do not need masking at inference time. The purpose of masking is that you prevent the decoder state from attending to positions that correspond to tokens "in the future", i.e., those that will not be known at the inference time, because they will not have been generated yet.

At inference time, it is no longer a problem because there are no tokens from the future, there have not been generated yet.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.