I am implementing a custom algo inspired by NMT architecture BUT in the decoder, if Query = target language then the "value" should also be the same thing right ? Only the "key" should be the encoder output ( encoded source language ). After much heartburn i have made peace with the fact that a "query" is what you are trying to find out and the "key" is some sort of index FOR the "values", from which you choose your answer (based on the best score generated by the attention algo ). So if i want to convert French to English, and my encoder is encoding French, then my query is got to be in English and so the values must be in English right ? why does the TF code ( NMT tutorial ) take the French encoding as the KEY and the VALUE ??

OR is the interpretation that since the query ( masked input , English ) and the key ( encoded French ) are first "dot producted" ( sorry ) together, the "values" are in fact being learnt during the training based on the loss calculated by difference between input context ( English ), so far and the next predicted word ? and during inference the "key-values" are now in the form of a French-English dictionary ( a very smart dict at that which gives nearest word, based on context ) ?


1 Answer 1


Note three things:

  1. The output of the encoder is not English but just what the decoder needs from the source sentence to generate the translation.

  2. Only the first decoder layer receives the target language token embeddings. The following layers receive the output of the previous layer.

  3. There are matrix multiplications before the dot products, which can project their inputs to completely different representation spaces.

So, to answer your question:

Neither of your interpretations is correct. The keys, values and queries are not in an "English representation space" nor in a "French representation space". Keys, vectors and queries are vectors in representation spaces that have been learned by the network during training. These representation spaces are not necessarily interpretable by a human, they were learned just to lower the loss at the task the model was trained in (i.e. to translate).

As an example of what I am trying to convey, please consider Transformer models trained for multilingual machine translation. These models can receive many different languages as input and generate translations in many different languages. These models learn to represent information in a way that makes it possible to translate properly between those languages (i.e. to minimize the loss they have been trained on). The same happens in a non-multilingual machine translation Transformer.

Actually, there are many scientific papers trying to understand what kind of information is encoded at the output of each layer in Transformer models.

  • $\begingroup$ sorry, but none of the points you are making are either the correct interpretation of my query OR a good answer ..my query is simple ..there HAS to be an explain-to-a-10-year-old interpretation of query, key and value construct ..and if everyone ( please dont take this personally ) just wants to obfuscate matters further, i am guessing this concept still hasn't been fully understood $\endgroup$ Jun 27, 2023 at 3:09
  • 1
    $\begingroup$ Sorry that I did not convey it clearly enough to be understood, but the point is that people think about neural network interpretability from a symbolic point of view, while that is not how it works. We certainly know how attention works in terms of keys, values and queries (see this), but it acts over a learned representation space, not English, nor French. Neural network interpretability is an active field of research, so you can hardly get the simple answer you request. I will leave my answer here in case it is useful to somebody else $\endgroup$
    – noe
    Jun 27, 2023 at 7:42
  • $\begingroup$ I added some extra information to try to make the answer more clear with an example. $\endgroup$
    – noe
    Jun 27, 2023 at 8:01
  • 1
    $\begingroup$ thanks for the additional info. It makes things a little clearer but i hope you dont mind if i wait till i choose this as the accepted answer..thanks again ! $\endgroup$ Jun 27, 2023 at 8:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.