6
$\begingroup$

I'm working through Attention is All you Need, and I have a question about masking in the decoder. It's stated that masking is used to ensure the model doesn't attend to any tokens in the future (not yet predicted), so it can be used autoregressively during inference.

I don't understand how masking is used during inference. When the encoder is given an unseen sample with no ground truth output or prediction, it seems to me that there is nothing to mask, since there aren't any output tokens beyond what the decoder has already produced. Is my understanding of masking correct?

Thanks!

Improve this question
$\endgroup$
2
  • $\begingroup$ Welcome to the forum! Your question is not clear. Please add some context. $\endgroup$
    – Peter
    Commented Aug 26, 2020 at 18:01
  • $\begingroup$ Okay, I tried to clarify it a bit. If it still is unclear, are there any clarifying questions you could ask? $\endgroup$
    – David Rein
    Commented Aug 26, 2020 at 20:26

1 Answer 1

Reset to default
3
$\begingroup$

The trick is that you do not need masking at inference time. The purpose of masking is that you prevent the decoder state from attending to positions that correspond to tokens "in the future", i.e., those that will not be known at the inference time, because they will not have been generated yet.

At inference time, it is no longer a problem because there are no tokens from the future, there have not been generated yet.

Improve this answer
$\endgroup$
4
  • $\begingroup$ This is wrong, see: ai.stackexchange.com/questions/16516/… You still need to apply masks in the decoder for all tokens except the final one, just like during training. $\endgroup$
    – Simon Boehm
    Commented Jul 17, 2022 at 19:59
  • 1
    $\begingroup$ It depends on how you use that decoder. If you start generating an empty string (like in MT), you only need to remember the past states and always when you do the self-attention, you always have the previous states and nothing to mask out. When you generate using a GPT-like model starting from a prompt, of course, you need to the masking for the prompt part. Also if you do not cache the previous hidden states and you want to recompute them in every generation step, you need to do the masking (which is what the answer you refer to assumes). $\endgroup$
    – Jindřich
    Commented Jul 18, 2022 at 9:47
  • $\begingroup$ @SimonBoehm Are you sure? :) Why do you need to "mask" even if it's your own prediction, not a given answer(ground truth)? And your link is not about the original transformer, which the OP is talking about. $\endgroup$
    – starriet 차주녕
    Commented Apr 27, 2023 at 7:03
  • $\begingroup$ youtube.com/… $\endgroup$
    – Miku Ghoul
    Commented Mar 5 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.