3
$\begingroup$

I am referring to seq2seq model's example code in keras (https://github.com/fchollet/keras/blob/master/examples/addition_rnn.py). The model is :

model = Sequential()

model.add(RNN(HIDDEN_SIZE, input_shape=(MAXLEN, len(chars))))

model.add(RepeatVector(DIGITS + 1))
for _ in range(LAYERS):
    model.add(RNN(HIDDEN_SIZE, return_sequences=True))

model.add(TimeDistributed(Dense(len(chars))))
model.add(Activation('softmax'))

In this model we are passing the encoded input vector from encoder's last state to each time step in the decoder.

Now we are not passing any other input to the decoder except the encoded input vector, but in all seq2seq models we pass output sequence also (time delayed) with the encoded input.

How is this a valid seq2seq model? To my surprise it works well. How does this works?

$\endgroup$
  • $\begingroup$ I think it works because it is the simplest task. $\endgroup$ – xxx222 Mar 25 '17 at 17:12
3
$\begingroup$

The original Seq2Seq paper uses the technique of passing the time delayed output sequence with the encoded input, this technique is termed teacher forcing.

enter image description here

There exists a simplified architecture in which fixed length encoded input vector is passed to each time step in decoder (analogy-wise, we can say, decoder peeks the encoded input at each time step).

The paper "Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation" combines these both techniques (so it passes encoded input vector along with time delayed output sequence as inputs to decoder).

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.