0
$\begingroup$

The original seq2seq paper reversed the input sequence and cited multiple reasons for doing so. See: Why does LSTM performs better when the source target is reversed? (Seq2seq)

But when using attention, is there still any benefit to doing this? I imagine since the decoder has access to the encoder hidden states at each time step, it can learn what to attend to and the input can be fed in the original order.

$\endgroup$

1 Answer 1

1
$\begingroup$

No, this trick is only relevant to the vanilla sequence-to-sequence models. It is exactly as you say. The decoder gets the information from the encoder via the attention mechanism, so there is no way the long distance between source and target words could block the capacity of the RNN state. Also, the encoder is typically bi-directional, which means there are two RNNs each of them processing the source sentence in a different direction. As a result, each encoder state contains information about the complete source sentence, one half in the left-side context, the other in the right-side context.

With today's Transformer models, it does not make any sense because Transformers treat the input as an unordered set and the only way it knows about the token order is the position embeddings (if the position embeddings are correct, you can permute the input tokens randomly).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.