Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
The original seq2seq paper reversed the input sequence and cited multiple reasons for doing so. See: Why does LSTM performs better when the source target is reversed? (Seq2seq)
But when using attention, is there still any benefit to doing this? I imagine since the decoder has access to the encoder hidden states at each time step, it can learn what to attend to and the input can be fed in the original order.
No, this trick is only relevant to the vanilla sequence-to-sequence models. It is exactly as you say. The decoder gets the information from the encoder via the attention mechanism, so there is no way the long distance between source and target words could block the capacity of the RNN state. Also, the encoder is typically bi-directional, which means there are two RNNs each of them processing the source sentence in a different direction. As a result, each encoder state contains information about the complete source sentence, one half in the left-side context, the other in the right-side context.
With today's Transformer models, it does not make any sense because Transformers treat the input as an unordered set and the only way it knows about the token order is the position embeddings (if the position embeddings are correct, you can permute the input tokens randomly).
