I do understand that dot product conveys the meaning of similarity in a vector space. At the same time it looks like during the training process we are learning the weights( or how much attention) each token in a sequence should put into other tokens. So the question is why is that important to have scalar value from the dot product when we could just learn a corresponding(a bigger for example) weight during training.
One of the reasons behind my confusion comes from a common example/explanation of how attention/self-attention works: model is able to 'understand' which other word token
it is connected to.
it(animal) is too tired, or it(road) is too wide
In that scenario dot product between
animal/road should be almost the same( given reasonable initial words' embeddings). And we are still able to learn proper weights.
- If and in what way dot product helps to learn attention weights?
- maybe there are other benefits that dot product brings us?