3
$\begingroup$

I'm currently reading this paper : https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf

Here, the author mentioned that LSTM learns better when the order of the source target is reversed. "Mapping (c,a'), (b,b'), (a,a') (a',b',c' is a translation of a,b,c each) is better." but I don't know why reversing the sentence makes it easy for SGD to map input n output sequence better.

$\endgroup$
1
  • 1
    $\begingroup$ Section 3.3 of that paper attempts to give an explanation: "While we do not have a complete explanation to this phenomenon, we believe that it is caused by the introduction of many short term dependencies to the data set", etc. $\endgroup$ – liangjy Apr 17 '17 at 14:51
2
$\begingroup$

Also the paper said, we do not have a complete explanation to this phenomenon.

But here is kinda explanation about that.

While we do not have a complete explanation to this phenomenon, we believe that it is caused by the introduction of many short term dependencies to the dataset.

Normally, when we concatenate a source sentence with a target sentence, each word in the source sentence is far from its corresponding word in the target sentence.

As a result, the problem has a large “minimal time lag” [17].

By reversing the words in the source sentence, the average distance between corresponding words in the source and target language is unchanged.

However, the first few words in the source language are now very close to the first few words in the target language, so the problem’s minimal time lag is greatly reduced.

Thus, backpropagation has an easier time “establishing communication” between the source sentence and the target sentence, which in turn results in substantially improved overall performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.