By far, I find this tutorial on self-attention the most digestible (https://peterbloem.nl/blog/transformers)
Still, I got a question from reading there, hopefully, you guys can help me out
- Are the component values of the input vectors updated throughout the training? (or the learned parts are the weight matrix Wq, Wk, Wv)?
In the blog, the first part said:
Since we are learning what the values in 𝐯t should be, how "related" two words are is entirely determined by the task. In most cases, the definite article the is not very relevant to the interpretation of the other words in the sentence; therefore, we will likely end up with an embedding 𝐯the that have a low or negative dot product with all other words
So I assume the components of v_the will be learned. Say, if it has 4 component values, like v_the = [ 1, 2, 3, 4], then after certain epochs of training, it will become like v_the = [0, 0.1, 0.01, 0.001]
But then a bit further down, when he started introducing W_q, W_k, and W_v, he said:
This gives the self-attention layer some controllable parameters and allows it to modify the incoming vectors to suit the three roles they must play.
So now it seems like we just keep the starting values of the input vectors in tact, and the training process will just update the corresponding W_q, W_k, W_v
Hence the question above.