0
$\begingroup$

It is common knowledge in the field of Deep Learning that the most powerful Recurrent architecture is the sequence-to-sequence, or seq2seq, for pretty much any task (to time series forecasts, to machine translation, to text generation).

Why? What are the underlying mathematical reasons for which an LSTM encoder-decoder architecture would outperform more canonical RNNs? Is it in the generation of dense latent representations? Is it about the comparatively higher number of parameters? Any hint is appreciated.

$\endgroup$
  • $\begingroup$ Where your simple RNN and seq2seq models (where you observed the huge improvement) of the same total number of parameters or did the seq2seq have more? $\endgroup$ – ncasas Dec 12 '19 at 16:21
0
$\begingroup$

Encoder-decoder architectures are not simply "more powerful" than a mere LSTM. LSTMs can't be used (in their standard configuration) for general sequence transduction tasks. On the other hand, encoder-decoder architectures are conditional autoregressive models, that is, they generate a sequence element by element conditioning on another sequence. This difference is what justifies the rather different use cases for simple LSTMs and encoder-decoder architectures.

Among LSTM-based encoder-decoder architectures, we should distinguish different types by the way information is passed from the encoder to the decoder. The simplest form is to simply pass the last hidden state of the encoder LSTM to the first decoder LSTM; this implies that all the information from the input sequence is "compressed" into a fixed-length vector, which is known to be an information bottleneck. More complex forms include using attention mechanisms, where the encoder hidden states at each time step are combined into a different previous context vector for each decoder LSTM; here there is no bottleneck, and normally their results are much better.

$\endgroup$
  • $\begingroup$ No question on attention models, they are another level of DL. What puzzles me is this: I happened to use either seq2seq and "simple" RNNs for time series multistep forecast, and I have observed a huge improvement in performance with seq2seq. I'm trying to fully understand the technical, mathematical reasons for that. $\endgroup$ – Leevo Nov 30 '19 at 8:46
  • $\begingroup$ Assuming the same input time window and hyperparameters for the simple RNN and the seq2seq model, the seq2seq is basically doubling the number of trainable parameters, and therefore doubling the (potential) model capacity. Where your simple RNN and seq2seq models (where you observed the huge improvement) of the same capacity or did the seq2seq have more capacity? $\endgroup$ – ncasas Dec 1 '19 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.