I'm not quite sure how's the decoder output is flattened into a single vector. As from my understanding, if we input the encoder with a length N sentence, it's output is N x units (e.g. N x 1000), and we input the decoder with a length M sentence, the output of the decoder will give us M x units output. M is not fixed (M should be the length of the decoder's raw input) and will change during the different steps of inference. How do we go from here to a single vector? enter image description here Screen shot from "Attention is all you need"


1 Answer 1


I'm not quite sure how's the decoder output is flattened into a single vector

That's the thing. It isn't flattened into a single vector. The linear transformation is applied to all $M$ vectors in the sequence individually. These vectors have a fixed dimension, which is why it works.

  • 1
    $\begingroup$ I'm still not quite sure I understand, let's take for example a language model. 1. when training, we train on the entire sentence at once? input W_0-Wn-1, output W_1-Wn? 2. when performing inference (e.g. sentence generation), we need to generate each word in a sequential way. So each new step we again predict all the previous words that we already generated(which should not change due to masking)? $\endgroup$
    – Ian
    Commented May 20, 2020 at 14:08
  • $\begingroup$ I'd recommend looking at this blog to better understand how text generation works with a transformer: jalammar.github.io/illustrated-transformer In short, encoding can be done in one go. Decoding happens sequentially. Decode first token, feed back prediction into decoder, then decode next token, and so on, until the stopping criterion is met. $\endgroup$ Commented May 20, 2020 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.