Is there utility in using different tokens for end-of-sentence, start-of-sentence, and padding for autoregressive sequence modeling (i.e. text generation)? Or can I use the same token for all of them?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Please consider upvoting and accepting the answer or, alternatively, describe why you consider it not to be correct or not clear enough. $\endgroup$– noeCommented Feb 24, 2023 at 18:48
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
Normally start-of-sequence is the same as end-of-sequence, that is, usually you use the end-of-sequence token to mark the start of the sequence.
The padding token is usually different, because that way you can easily compute the masks to use to mark which tokens should be ignored. end of sequence and start of sequence positions should not be ignored, so it's useful to have different padding than start/end of sequence.
-
$\begingroup$ I understand the rationale for the EOS token, as the model neads to learn when to terminate a sentence. But I am beginning to doubt the utility of the SOS token. It's not used in the encoder of traditional seq2seq models, and my testing it out in sentence autoencoders did also not yield any difference. Any representation that is passed between neural nets is the least diluted at the beginning due to the autoregressive nature of the problem. I agree with the rest of your answer, thank you for the reply. $\endgroup$ Commented Feb 24, 2023 at 19:15
-
$\begingroup$ SOS token is normally used in encoder-decoder architectures, as initial input of the decoder. $\endgroup$– noeCommented Feb 24, 2023 at 19:32
-
$\begingroup$ I am aware of that, but I am not sure whether it is strictly necessary. $\endgroup$ Commented Feb 24, 2023 at 19:51
-
$\begingroup$ Well, you need some initial placeholder token, either a dedicated SOS token, or the EOS token or any other token you choose as first token to every sequence received as input by the decoder. $\endgroup$– noeCommented Feb 24, 2023 at 20:02