In transformers, there is a phase for rasidual connection, where the queries and the output from the attention are add and normalize. Can one please give some advise to the motivation of it? Or maybe I get it wrong? It seems to me that the values shouldn't come from the encoder, the values are the vector that we want to have attention on. And if so. We should have add and normalize the values from the previous state and not the queries... I'm confused..


1 Answer 1


In order to understand how the attention block works maybe this analogy helps: think of the attention block as a Python dictionary, e.g.

keys =   ['a', 'b', 'c']
values = [2, 7, 1]
attention = {keys[0]: values[0], keys[1]: values[1], keys[2]: values[2]}
queries = ['c', 'a']
result = [attention[queries[0]], attention[queries[1]]]

In the code above, result should have value [1, 2].

The attention from the transformer works in a similar way, but instead of having hard matches, it has soft maches: it gives you a combination of the values weighting them according to how similar their associated key is to the query.

In the encoder-decoder attention blocks, the keys and values are the encoder output and the queries are the decoder states. The logic behind this is that the new hidden states in the decoder are a combination of the states of the encoder (i.e. the source sentence representations) weighted by their similarity (scaled dot product) with the partially decoded target sentence representations.

The residual connection and normalization elements are common processing steps applied after each multi-head attention and positional feedforward layer, independently from the origin of queries, keys, and values used in it:

enter image description here

P.S.: some parts of this answer are reused from my answer to another question.

  • $\begingroup$ thanks for the fast replay! I'm still having two questions: 1.If we imagine that we don't have skip connection then the output of the decoder are values from the original sentence - queries are only affecting where the attention should focus in the values vector - is this really should be the values from the encoder and not from the previous state of the decoder? 2.If we have skip connection, then we are getting is a combination of the queries and the values which not make sense,since the values are representing the original sentence and the queries are representing the decoder state. $\endgroup$
    – eran
    Dec 5, 2020 at 18:58
  • $\begingroup$ About 1 and 2, I think it is not practical to think in terms of "should be" or "shouldn't be", only what it is, and that a neural architecture is not about "making sense", but about getting good results. Normally, neural architectures are the result of a process where the beginning may be something that "made sense", but when it is tested it does not work, and then some modifications are applied in order to mitigate issues (e.g. exploding norms --> introduce normalization, vanishing gradients when many layers are stacked --> add residual connection, etc). $\endgroup$
    – noe
    Dec 5, 2020 at 19:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.