0
$\begingroup$

I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that one will understand their purpose after reading about how they are used for attention. I believe I understand what they do, but I'm unsure about how they are created.

I'm aware that they come from the multiplication of the input vector by three corresponding weights, but I'm not sure how these weights are derived. Are they chosen at random and trained like a standard neural network, and if so how if there's no predefined attention data in the training corpus?

I'm very new to this, so I hope everything I'm saying makes sense. If I've got something completely wrong, please tell me!

$\endgroup$
1
$\begingroup$

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous layers. In encoder-decoder attention, the queries are decoder states from the previous layer, keys and values and the encoder states.

In Equation 1 of the Attention is all you need paper, these are just parameters that come from outside:

enter image description here

It just says, what do you do, if get some queries, keys, and values from somewhere outside. This also the case of the unnumbered equation on top of page 5. Here, you only project the keys, queries, and values for the heads of multiple attentions.

enter image description here

Here $W_i^Q$, $W_i^K$, and $W_i^V$ are learnable parameters and they learned by the standard back-propagation algorithm.

Note that although keys and values (and queries in the self-attention) are equal only at the input of the $\text{Multihead}$ function. The $\text{Attention}$ function already gets the projected states.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you so much! That really helps, but at what stage are the weights trained? And wouldn't that backpropagation interfere with that of the rest of the system? Is there not other backpropagation elsewhere? $\endgroup$ – Arctic_Hen7 Feb 17 at 19:34
1
$\begingroup$

Q, K, V vectors are trained with standard backpropagation. All trainable parameters are initialized at random, and then adjusted step by step with a Gradient Descent algorithm.

Surprisingly, they are trained just as any standard ANN! It's pretty amazing what they can achieve with such a classical trick.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.