1
$\begingroup$

By far, I find this tutorial on self-attention the most digestible (https://peterbloem.nl/blog/transformers) 

Still, I got a question from reading there, hopefully, you guys can help me out 

  • Are the component values of the input vectors updated throughout the training? (or the learned parts are the weight matrix Wq, Wk, Wv)? 

In the blog, the first part said:

Since we are learning what the values in 𝐯t should be, how "related" two words are is entirely determined by the task. In most cases, the definite article the is not very relevant to the interpretation of the other words in the sentence; therefore, we will likely end up with an embedding 𝐯the that have a low or negative dot product with all other words

So I assume the components of v_the will be learned. Say, if it has 4 component values, like  v_the = [ 1, 2, 3, 4], then after certain epochs of training, it will become like v_the = [0, 0.1, 0.01, 0.001]

But then a bit further down,  when he started introducing W_q, W_k, and W_v, he said: 

This gives the self-attention layer some controllable parameters and allows it to modify the incoming vectors to suit the three roles they must play.

So now it seems like we just keep the starting values of the input vectors in tact, and the training process will just update the corresponding W_q, W_k, W_v 

Hence the question above.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The training of a self-attention layer will result in the update of the $W$ matrices and the gradient being propagated back to the previous layer.

At the end of the self-attention blocks, the back-propagated gradient will arrive to the embedded vectors, which will also be updated.

As personal advice, I would suggest that you don't try to understand why self-attention works, just how it works. The analogies made in the linked post about the embeddings of "the", "cat" and "walk" are nonsense in my opinion. First, nowadays most neural text processing models work at the subword level, not at the word level. Also, self-attention layers are stacked, so the token identities are lost after the first layer (unless you are training something like a masked language model where you predict the very same input tokens at the same positions). *

Improve this answer
$\endgroup$
13
  • $\begingroup$ Thanks @noe, I understand the subword part you mention, like BPE or WordPiece, but before your answer, I was wondering the value of the incoming vectors (before entering self-attention layer) are trained (updated) or not. Now it's clearer, they are not, are they? $\endgroup$
    – EyeQ Tech
    Commented Feb 26, 2023 at 0:34
  • $\begingroup$ It depends on where those incoming vectors come from. In the first self-attention layer, they come from the embeddings; in that case, they are updated. In the other self-attention layers, they come from the outputs of previous self-attention blocks, so the gradients are propagated backward. $\endgroup$
    – noe
    Commented Feb 26, 2023 at 8:21
  • $\begingroup$ Hi @noe, tks for your comment. So I misunderstood the author. OK, so the next question: if the weight matrices W_Q, W_K, W_V are trained, then theoretically we just keep the weight of input embeddings the same (randomly initiated values), then why the need to train them (them = input embeddings)? $\endgroup$
    – EyeQ Tech
    Commented Mar 4, 2023 at 11:12
  • $\begingroup$ The embeddings are trained, they are not kept the same $\endgroup$
    – noe
    Commented Mar 4, 2023 at 12:33
  • $\begingroup$ Thanks @noe, I just ask that for theoretical clarity: if ... then theoretically... (above comment), there's no need to train input embeddings, is it true? $\endgroup$
    – EyeQ Tech
    Commented Mar 4, 2023 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.