Timeline for What are the inputs to the first decoder layer in a Transformer model during the training phase?
Current License: CC BY-SA 4.0
4 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 5, 2021 at 23:02 | vote | accept | djvaroli | ||
| Feb 5, 2021 at 20:59 | comment | added | noe |
Yes, you are right, the sequence should actually be <start> I am fine. Note that in some implementations, instead of using a specific token for <start>, the <eos> token is reused, and in those cases the sequence would be <eos> I am fine. This is the case of fairseq; you can check their implementation, which is quite clear.
|
|
| Feb 5, 2021 at 18:14 | comment | added | djvaroli | Thanks for your answer! Am I correct in understanding then that the Transformer will always output a sequence (I think that's probably self-explanatory but for some reason, I wasn't sure about it). My follow-up question would be, how is it that the decoder input and decoder output sequences will be of different lengths in the example you gave above? | |
| Feb 5, 2021 at 14:35 | history | answered | noe | CC BY-SA 4.0 |