# Tensorflow diagram for attention mechanism

I was reading the tutorial from tensorflow on the transformer model, however, when they explain the transformer model, they display such a picture :

which I don't understand. What do the ingoing arrow and outgoing arrow mean? What do the bold vertical line and the normal horizontal lines mean? Why is there just one array for (key,value)?

This is a diagram of the attention layer that appears in the English version of the Tensorflow Transformer tutorial (other languages do not have this figure).

The ingoing arrows are inputs to the attention layer. Query is a sequence of vectors (each square in Query represents one of the vectors). Key is a sequence of vectors (each square in Key represents one of the vectors). Value is the same as Key.

Inside the attention layer are multiple attention heads (right side of the figure above), and each head uses a projection Key and a projection of Query to find N weights in [0, 1] that are used to weight-sum Value into a single vector (left side of the figure above). This happens for each vector of Query.

Each of the purple columns is the collection of N weights to be applied to the Value vectors to compute the output at the column position. Therefore the bold vertical lines separate the computations for each position of the Query, while the horizontal lines represent the positions of the weights applied to Value vectors. The outgoing arrows are the outputs of the attention layer.

• probably your explanation is good but I still don't get it. In this example N=3? In each purple square there is a scalar? Each scalar is multiplied by the corresponding key, value in the horizontal line? This would give a column (scalar1×key_1, scalar_2*key_2, scalar_3*key_3) then this is "dot producted" with the corresponding query at the bottom of the column? Commented Sep 1, 2023 at 16:00
• N=3? → yes. In each purple square there is a scalar? → yes. Each scalar is multiplied by the corresponding key, value in the horizontal line → no, each scalar multiplies one of the vectors of Value, then the results are added together (see left side of my figure), giving the output that that column's position
– noe
Commented Sep 1, 2023 at 16:15
• You can also check the math in the original paper.
– noe
Commented Sep 1, 2023 at 16:16