In the original paper, the author says that the annotation are the concatenation fo the forward states and the backward states at each time step.

enter image description here

In the tensorflow implementation (memory param), the memory field is said to be populated with the output (not hidden state) of an RNN encoder.

What am I missing?


1 Answer 1


in case you havent yet found the answer, tensorflows default attention implementation doesn't perform bi-directional encoding hence you dont see the concatenation (whereas in the paper , its clearly mentioned) ..i am guessing we need to include b-directional rnn's explicitly to mimic the paper. For further proof look at line 788 https://github.com/tensorflow/tensorflow/blob/r1.3/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py

the comment below the function definition clearly tells you that "Then it runs an RNN to encode embedded encoder_inputs into a state vector. It keeps the outputs of this RNN at every step to use for attention later. Next, it embeds decoder_inputs..."

hope that helped


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.