0
$\begingroup$

As this question says:

In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix:

enter image description here

The reason why is stated that this constrains the distribution of the weights of the output to have a standard deviation of 1.

My question is why don't we do the same after multiplying to $V$(values) for the same reason?

$\endgroup$

1 Answer 1

0
$\begingroup$

I mean you could. The standardization here is to control the variance of the dot product as otherwise, since the softmax exponentiates, the weightings of the softmax are likely to be very sparse (i.e. a weight of near $1$ applied to the largest dot product value). This is not very helpful as the gradients will be extremely small for values with softmax weights of $0$.

As for normalizing the products, you can do so, and I believe the architecture does include Normalization layers for this purpose.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.