Why do we need to add START <s> + END </s> symbols when using Recurrent Neural Nets for Sequence-to-Sequence Models?

Question

In the Sequence-to-Sequence models, we often see that the START (e.g. <s>) and END (e.g. </s>) symbols are added to the inputs and outputs before training the model and before inference/decoding unseen data.

E.g. http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

SOS_token = 0
EOS_token = 1


class Lang:
    def __init__(self, name):
        self.name = name
        self.word2index = {}
        self.word2count = {}
        self.index2word = {0: "SOS", 1: "EOS"}
        self.n_words = 2  # Count SOS and EOS

    def addSentence(self, sentence):
        for word in sentence.split(' '):
            self.addWord(word)

    def addWord(self, word):
        if word not in self.word2index:
            self.word2index[word] = self.n_words
            self.word2count[word] = 1
            self.index2word[self.n_words] = word
            self.n_words += 1
        else:
            self.word2count[word] += 1

• Is there a technical definition or academic explanation of why that is necessary?

• Or does that need to add the END symbol only applies to Natural Language Processing task where the sentence generation needs to end?

• But what does the START symbol do? Other than to take the initial state where the trained network will start inferrring.

Answer 1

Because of the encoder-decoder structure. The encoder reads the input sequence to construct an embedding representation of the sequence. Terminating the input in an end-of-sequence (EOS) token signals to the encoder that when it receives that input, the output needs to be the finalized embedding. We (normally) don't care about intermediate states of the embedding, and we don't want the encoder to have to guess as to whether or not the input sentence is complete or not.

The EOS token is important for the decoder as well: the explicit "end" token allows the decoder to emit arbitrary-length sequences. The decoder will tell us when it's done emitting tokens: without an "end" token, we would have no idea when the decoder is done talking to us and continuing to emit tokens will produce gibberish.

The start-of-sequence (SOS) token is more important for the decoder: the decoder will progress by taking the tokens it emits as inputs (along with the embedding and hidden state, or using the embedding to initialize the hidden state), so before it has emitted anything it needs a token of some kind to start with. Hence, the SOS token.

Additionally, if we're using a bidirectional RNN for the encoder, we're definitely going to want to use both SOS and EOS tokens since the SOS token will signal to the reversed-input layer when the input is complete (otherwise, how would it know?).