Principle behind seq2seq model's example in keras?

Question

I am referring to seq2seq model's example code in keras (https://github.com/fchollet/keras/blob/master/examples/addition_rnn.py). The model is :

model = Sequential()

model.add(RNN(HIDDEN_SIZE, input_shape=(MAXLEN, len(chars))))

model.add(RepeatVector(DIGITS + 1))
for _ in range(LAYERS):
    model.add(RNN(HIDDEN_SIZE, return_sequences=True))

model.add(TimeDistributed(Dense(len(chars))))
model.add(Activation('softmax'))

In this model we are passing the encoded input vector from encoder's last state to each time step in the decoder.

Now we are not passing any other input to the decoder except the encoded input vector, but in all seq2seq models we pass output sequence also (time delayed) with the encoded input.

How is this a valid seq2seq model? To my surprise it works well. How does this works?

Answer 1

The original Seq2Seq paper uses the technique of passing the time delayed output sequence with the encoded input, this technique is termed teacher forcing.

There exists a simplified architecture in which fixed length encoded input vector is passed to each time step in decoder (analogy-wise, we can say, decoder peeks the encoded input at each time step).

The paper "Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation" combines these both techniques (so it passes encoded input vector along with time delayed output sequence as inputs to decoder).