I was reading the tutorial from tensorflow on the transformer model, however, when they explain the transformer model, they display such a picture :

enter image description here which I don't understand. What do the ingoing arrow and outgoing arrow mean? What do the bold vertical line and the normal horizontal lines mean? Why is there just one array for (key,value)?


1 Answer 1


This is a diagram of the attention layer that appears in the English version of the Tensorflow Transformer tutorial (other languages do not have this figure).

The ingoing arrows are inputs to the attention layer. Query is a sequence of vectors (each square in Query represents one of the vectors). Key is a sequence of vectors (each square in Key represents one of the vectors). Value is the same as Key.

enter image description here

Inside the attention layer are multiple attention heads (right side of the figure above), and each head uses a projection Key and a projection of Query to find N weights in [0, 1] that are used to weight-sum Value into a single vector (left side of the figure above). This happens for each vector of Query.

Each of the purple columns is the collection of N weights to be applied to the Value vectors to compute the output at the column position. Therefore the bold vertical lines separate the computations for each position of the Query, while the horizontal lines represent the positions of the weights applied to Value vectors. The outgoing arrows are the outputs of the attention layer.

  • $\begingroup$ probably your explanation is good but I still don't get it. In this example N=3? In each purple square there is a scalar? Each scalar is multiplied by the corresponding key, value in the horizontal line? This would give a column (scalar1×key_1, scalar_2*key_2, scalar_3*key_3) then this is "dot producted" with the corresponding query at the bottom of the column? $\endgroup$
    – edamondo
    Sep 1 at 16:00
  • $\begingroup$ N=3? → yes. In each purple square there is a scalar? → yes. Each scalar is multiplied by the corresponding key, value in the horizontal line → no, each scalar multiplies one of the vectors of Value, then the results are added together (see left side of my figure), giving the output that that column's position $\endgroup$
    – noe
    Sep 1 at 16:15
  • $\begingroup$ You can also check the math in the original paper. $\endgroup$
    – noe
    Sep 1 at 16:16
  • $\begingroup$ @edamondo did my answers to your questions clarify your doubts? $\endgroup$
    – noe
    Sep 6 at 7:48
  • $\begingroup$ I went back and forth but couldn't really understand. I think it would have been useful if the authors had drew a pictures with the concrete values inside. However thank you for taking the time answering! $\endgroup$
    – edamondo
    Sep 8 at 12:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.