Why seq2seq models are superior to simple LSTMs?

Question

It is common knowledge in the field of Deep Learning that the most powerful Recurrent architecture is the sequence-to-sequence, or seq2seq, for pretty much any task (to time series forecasts, to machine translation, to text generation).

Why? What are the underlying mathematical reasons for which an LSTM encoder-decoder architecture would outperform more canonical RNNs? Is it in the generation of dense latent representations? Is it about the comparatively higher number of parameters? Any hint is appreciated.

Answer 1

Encoder-decoder architectures are not simply "more powerful" than a mere LSTM. LSTMs can't be used (in their standard configuration) for general sequence transduction tasks. On the other hand, encoder-decoder architectures are conditional autoregressive models, that is, they generate a sequence element by element conditioning on another sequence. This difference is what justifies the rather different use cases for simple LSTMs and encoder-decoder architectures.

Among LSTM-based encoder-decoder architectures, we should distinguish different types by the way information is passed from the encoder to the decoder. The simplest form is to simply pass the last hidden state of the encoder LSTM to the first decoder LSTM; this implies that all the information from the input sequence is "compressed" into a fixed-length vector, which is known to be an information bottleneck. More complex forms include using attention mechanisms, where the encoder hidden states at each time step are combined into a different previous context vector for each decoder LSTM; here there is no bottleneck, and normally their results are much better.