1
$\begingroup$

It is common knowledge in the field of Deep Learning that the most powerful Recurrent architecture is the sequence-to-sequence, or seq2seq, for pretty much any task (to time series forecasts, to machine translation, to text generation).

Why? What are the underlying mathematical reasons for which an LSTM encoder-decoder architecture would outperform more canonical RNNs? Is it in the generation of dense latent representations? Is it about the comparatively higher number of parameters? Any hint is appreciated.

Improve this question
$\endgroup$
1
  • $\begingroup$ Where your simple RNN and seq2seq models (where you observed the huge improvement) of the same total number of parameters or did the seq2seq have more? $\endgroup$ – noe Dec 12 '19 at 16:21
1
$\begingroup$

Encoder-decoder architectures are not simply "more powerful" than a mere LSTM. LSTMs can't be used (in their standard configuration) for general sequence transduction tasks. On the other hand, encoder-decoder architectures are conditional autoregressive models, that is, they generate a sequence element by element conditioning on another sequence. This difference is what justifies the rather different use cases for simple LSTMs and encoder-decoder architectures.

Among LSTM-based encoder-decoder architectures, we should distinguish different types by the way information is passed from the encoder to the decoder. The simplest form is to simply pass the last hidden state of the encoder LSTM to the first decoder LSTM; this implies that all the information from the input sequence is "compressed" into a fixed-length vector, which is known to be an information bottleneck. More complex forms include using attention mechanisms, where the encoder hidden states at each time step are combined into a different previous context vector for each decoder LSTM; here there is no bottleneck, and normally their results are much better.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ No question on attention models, they are another level of DL. What puzzles me is this: I happened to use either seq2seq and "simple" RNNs for time series multistep forecast, and I have observed a huge improvement in performance with seq2seq. I'm trying to fully understand the technical, mathematical reasons for that. $\endgroup$ – Leevo Nov 30 '19 at 8:46
  • $\begingroup$ Assuming the same input time window and hyperparameters for the simple RNN and the seq2seq model, the seq2seq is basically doubling the number of trainable parameters, and therefore doubling the (potential) model capacity. Where your simple RNN and seq2seq models (where you observed the huge improvement) of the same capacity or did the seq2seq have more capacity? $\endgroup$ – noe Dec 1 '19 at 17:57
0
$\begingroup$

They're not disjunctive, seq2seq models can use LSTMs in their architecture, i.e both encoder and decoder can be made of deep bi-directional LSTMs.

Seq2seq models are a type of architecture that's useful for tasks like Machine translation. LSTMs are just a building block of neural networks. It's not possible to compare relative performance of both. This is sort of like asking the question, why is a car better than its engine, or a house better than the internal plumbing inside.

Seq2seq models in their latest form now use totally new technology called the Transformer, and may not use LSTMs anymore which might be considered old fashioned! I guess that's another reason for any perceived 'superiority'

For why LSTMs are better than RNNs, there's a lot of information out there if you google for the 'vanishing/exploding gradient problem'.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.