1
$\begingroup$

I do understand that dot product conveys the meaning of similarity in a vector space. At the same time it looks like during the training process we are learning the weights( or how much attention) each token in a sequence should put into other tokens. So the question is why is that important to have scalar value from the dot product when we could just learn a corresponding(a bigger for example) weight during training.

One of the reasons behind my confusion comes from a common example/explanation of how attention/self-attention works: model is able to 'understand' which other word token it is connected to.

it(animal) is too tired, or it(road) is too wide

In that scenario dot product between it and animal/road should be almost the same( given reasonable initial words' embeddings). And we are still able to learn proper weights.

Questions:

  1. If and in what way dot product helps to learn attention weights?
  2. maybe there are other benefits that dot product brings us?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.