I'm not quite sure how's the decoder output is flattened into a single vector.
As from my understanding, if we input the encoder with a length N sentence, it's output is N x units (e.g. N x 1000), and we input the decoder with a length M sentence, the output of the decoder will give us M x units output. M is not fixed (M should be the length of the decoder's raw input) and will change during the different steps of inference.
How do we go from here to a single vector?
Screen shot from "Attention is all you need"
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
I'm not quite sure how's the decoder output is flattened into a single vector
That's the thing. It isn't flattened into a single vector. The linear transformation is applied to all $M$ vectors in the sequence individually. These vectors have a fixed dimension, which is why it works.
answered May 20, 2020 at 14:00
-
1$\begingroup$ I'm still not quite sure I understand, let's take for example a language model. 1. when training, we train on the entire sentence at once? input W_0-Wn-1, output W_1-Wn? 2. when performing inference (e.g. sentence generation), we need to generate each word in a sequential way. So each new step we again predict all the previous words that we already generated(which should not change due to masking)? $\endgroup$– IanMay 20, 2020 at 14:08
-
$\begingroup$ I'd recommend looking at this blog to better understand how text generation works with a transformer: jalammar.github.io/illustrated-transformer In short, encoding can be done in one go. Decoding happens sequentially. Decode first token, feed back prediction into decoder, then decode next token, and so on, until the stopping criterion is met. $\endgroup$ May 20, 2020 at 14:30