Self Attention vs LSTM with Attention for NMT

Question

I am trying to compare the

• A: Transformer-based architecture for Neural Machine Translation (NMT) from the Attention is All You Need paper, with

• B: an architecture based on Bi-directional LSTM's in the encoder coupled with a unidirectional LSTM in the decoder, which attends to all the hidden states of the encoder, creates a weighted combination and uses this along with decoder (unidirectional) LSTM output to produce final output word.

My question is what might be the advantages of Architecture A over B i.e. Self Attention vs LSTM's with attention?

I would imagine that Architecture A has a big advantage of having parallel computation compared to the sequential nature of computation in Architecture B.

Are there any other advantages? In particular, would architecture A have maximum path length advantages as described in the Attention is All you need paper?

Answer 1

The Transformer-based MT typically performs better than RNN-based MT in terms of translation quality. People used to claim that RNNs are better for low-resource language pairs, however, this is not true anymore with pre-trained models such as MASS or mBART.

The other advantage of Transformers is that at training time, they can be fully parallelized, whereas an RNN always processes the sentences sequentially. To compute the $n$-th state, you always need to wait until $n-1$-th is ready.

One disadvantage of the transformer decoder is that at every step it needs to attend to all previously decoded tokens, which makes the generation quadratic in theory (in practice, this can be parallelized quite well). When efficiency is a concern, it might be a good idea to combine a Transformer encoder with an RNN decoder.

Answer 2

You're essentially trying to compare the two revolutionary neural/attention-based MT models: Bahdanau 2015 and Vaswani 2017.

A first difference is that their attention weights formula is slightly different: Bahdanau uses a feed-forward network to essentially try to learn a similarity function. Vaswani uses a scaled dot product.

A second difference is that during encoder-decoder attention, Bahdanau compares one decoder vector to all encoder vectors. Vaswani, on the other hand, compares all decoder vectors to all encoder vectors.

A third difference: as Jindrich mentioned, transformers are faster to train because the encoder is parallel, not sequential. Apart from time, they also need fewer computations (about 100 times) than the state-of-the-art NMT models at the time the Vaswani paper was published (see table 2 on page 8) to obtain a competitive translation quality (BLEU). I don't know how this compares to the Bahdanau architecture, since Vaswani doesn't include it.

A fourth important difference between the two architectures is how they treat long-range dependencies. It is often said that Bahdanau solved Seq2Seq's information bottleneck problem by exposing all the encoder's hidden states to the decoder instead of only the final one. However, there are actually still two information bottlenecks present in Bahdanau, namely within the encoder and decoder respectively. Information from words at one end of the sentence has a hard time propagating all the way through to the other end. All of that information is crammed into the LSTM's hidden state, which becomes less and less sharp as it ripples through. In a transformer, the attention between two words is unaffected by the distance between them -- ceteris paribus, e.g. disregarding positional encoding. The dot product is the same dot product.

One final difference between the two, based on the above: Bahdanau essentially pre-processes word embeddings with a BiLSTM, performing attention over what that spits out. Vaswani performs attention over the embeddings directly, merely adding a modulation to the individual words (a pre-processing step where the words don't interact yet) to discern whether the same word is in a different position.