In the original paper, the author says that the annotation are the concatenation fo the forward states and the backward states at each time step.
In the tensorflow implementation (memory param), the memory field is said to be populated with the output (not hidden state) of an RNN encoder.
What am I missing?
in case you havent yet found the answer, tensorflows default attention implementation doesn't perform bi-directional encoding hence you dont see the concatenation (whereas in the paper , its clearly mentioned) ..i am guessing we need to include b-directional rnn's explicitly to mimic the paper. For further proof look at line 788 https://github.com/tensorflow/tensorflow/blob/r1.3/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py
the comment below the function definition clearly tells you that "Then it runs an RNN to encode embedded encoder_inputs into a state vector. It keeps the outputs of this RNN at every step to use for attention later. Next, it embeds decoder_inputs..."
hope that helped
