Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I was reading the tutorial from tensorflow on the transformer model, however, when they explain the transformer model, they display such a picture :
which I don't understand. What do the ingoing arrow and outgoing arrow mean? What do the bold vertical line and the normal horizontal lines mean? Why is there just one array for (key,value)?
This is a diagram of the attention layer that appears in the English version of the Tensorflow Transformer tutorial (other languages do not have this figure).
The ingoing arrows are inputs to the attention layer. Query is a sequence of vectors (each square in Query represents one of the vectors). Key is a sequence of vectors (each square in Key represents one of the vectors). Value is the same as Key.
Inside the attention layer are multiple attention heads (right side of the figure above), and each head uses a projection Key and a projection of Query to find N weights in [0, 1] that are used to weight-sum Value into a single vector (left side of the figure above). This happens for each vector of Query.
Each of the purple columns is the collection of N weights to be applied to the Value vectors to compute the output at the column position. Therefore the bold vertical lines separate the computations for each position of the Query, while the horizontal lines represent the positions of the weights applied to Value vectors. The outgoing arrows are the outputs of the attention layer.
