I was reading the tutorial from tensorflow on the transformer model, however, when they explain the transformer model, they display such a picture :
This is a diagram of the attention layer that appears in the English version of the Tensorflow Transformer tutorial (other languages do not have this figure).
The ingoing arrows are inputs to the attention layer. Query is a sequence of vectors (each square in Query represents one of the vectors). Key is a sequence of vectors (each square in Key represents one of the vectors). Value is the same as Key.
Inside the attention layer are multiple attention heads (right side of the figure above), and each head uses a projection Key and a projection of Query to find N weights in [0, 1] that are used to weight-sum Value into a single vector (left side of the figure above). This happens for each vector of Query.
Each of the purple columns is the collection of N weights to be applied to the Value vectors to compute the output at the column position. Therefore the bold vertical lines separate the computations for each position of the Query, while the horizontal lines represent the positions of the weights applied to Value vectors. The outgoing arrows are the outputs of the attention layer.