In the Vaswani 2017 paper introducing encoder-decoder transformers, the cross-attention step in the decoder is visualised as follows:


Because keys and values are always taken to be equal, this figure implies that the final encoder embeddings are used as keys and values, and the intermediate decoder embeddings are used as queries. Indeed, they write:

In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder.

What this means is that we have $m$ decoder embeddings coming in from the previous decoder block, yet counterintuitively, we don't have $m$ linear combinations of those decoder embeddings coming out of the current decoder block, but $m$ linear combinations of the encoder embeddings.

Although this is apparently not unheard of (Bahdanau 2015 sort of have the same thing by having the context vector, which is half of the recurrent input of the decoder, be a linear combination of the encoder embeddings), it has two very strange consequences:

  • Assuming a distinct tokeniser for the target language (which isn't uncommon), the decoder has a separate embedding matrix which it indexes as its very first processing step. Its static embeddings pertain to tokens in the target language. These embeddings flow into the first decoder block. Yet, after that one decoder block, we have gone from these static target-language embeddings to a linear combination of contextualised source-language embeddings. The values in the decoder's embedding matrix are essentially thrown away after one decoder block, having contributed to nothing more than one set of dot products.

  • The decoder essentially regurgitates the same vectors in each block.

    • In the encoder, a block starts out with $n$ embeddings. You then change these embeddings (with self-attention and an MLP). In the next block, you start with these changed embeddings, and change them again.
    • In the decoder, a block starts out with $m$ embeddings. Then you change them (with self-attention). Then you turn them into $m$ linear combinations of $n$ encoder embeddings. Then you change these (with an MLP). In the next block, you change these (with self-attention)... and then you go back to a linear combination of the $n$ old embeddings that you already transformed in the previous block. You keep circling back to replacing your work by (different linear combinations of) the same embeddings, rather than transforming them recursively.

User noe boiled this down to the "black-boxiness" of neural models in this thread: if it doesn't make sense, just assume you're wrong and that the machine is right.

[In a French-to-English model, the] keys, values and queries are not in an "English representation space" nor in a "French representation space". Keys, vectors and queries are vectors in representation spaces that have been learned by the network during training. These representation spaces are not necessarily interpretable by a human, they were learned just to lower the loss at the task the model was trained in (i.e. to translate).

This might handwave away the first concern (embeddings have "no language" even though we have different static embeddings for different languages), but the second concern requires an architectural motivation. Why do we regurgitate the encoder embeddings? Why don't we use the encoder vectors as queries to reweight a sequence of $m$ constantly evolving decoder embeddings used as keys and values? There must be a good motivation for doing it this way, right?

I have heard that some systems use deep encoders and shallow decoders. It isn't obvious to me that this isn't a side-effect of regurgitating the encoder embeddings. If you're not letting the decoder come up with vastly new embeddings in each block, there's no point in having a deep decoder.

  • $\begingroup$ Transformers are autoregressive. The decoder sequence is created incrementally. This implies that, at the beginning, there decoder input is just <bos>, so there is no sequence of $m$ decoder embeddings to reweight. $\endgroup$
    – noe
    Jan 8 at 15:23
  • $\begingroup$ @noe Of course there is a sequence of decoder embeddings to reweight. That sequence is whatever flows from one decoder block to the next. $m = 1$ at the start, append the autoregressively generated token and run with $m = 2$, then $m = 3$, ... $\endgroup$
    – Mew
    Jan 8 at 15:27
  • $\begingroup$ Wouldn't that lead to a constant output at the first step? $\endgroup$
    – noe
    Jan 8 at 15:36
  • $\begingroup$ @noe If the first token in the decoder is always the same, then yes, the first embedding would be the same for all encoder token sequences. But we weren't going to be using that token anyway in NMT. From there onward, you get dependencies on the output tokens in exactly the same way decoder self-attention is normally masked (only seeing leftward tokens). $\endgroup$
    – Mew
    Jan 8 at 15:41


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.