In self-attention, the attention for a word is calculated as:

$$ A(q, K, V) = \sum_{i} \frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}v^{<i>} $$

My question is why we sum over the Softmax*Value vectors. Doesn't this lose information about which other words in particular are important to the word under consideration?

In other words, how does this summed vector point to which words are relevant?

For example, consider two extreme scenarios where practically the entire output depends on the attention vector of word $x^{<t>}$, and one where it depends on the vector of word $x^{<t+1>}$. It's possible that $A(q, K, V)$ has the exact same values in both scenarios.


I guess there is an assumption that $A(q, K, V)$ represents similarity between q and the words in the sentence. Actually $A(q, K, V)$ is encoding the relationships from q to the words in the sentence.

The information about which other words in particular are important to the word under consideration has been already encoded by $q \cdot k^{<i>}$.

Suppose there is a sentence i love sushi, $v^{<i>}$ is a positionally-encoded word-embedding vector for each word, and the word under consideration $q = v^{<1>}$ which corresponds with love.

$v^{<0>} = i\\ v^{<1>} = love\\ v^{<2>} = sushi$

$scale(i)=\frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}$ quantifies how strong the connection is between love and $v^{<i>}$.

$\frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}v^{<i>}$ or $scale(i)v^{<i>}$ is scaling each vector $v^{<i>}$ proportionally based on how strongly related $v^{<i>}$ is to love.

Aggregating all the scaled vectors $\sum_{i} scale(i)v^{<i>}$ creates a new vector $A$, which is encoding the relationships from q to the words in the sentence.

enter image description here

The vector $A$ goes through the neural network and the back-propagation on the scaling and aggregation operations are how the Transformer learns the relations between q and the sentence.

  • $\begingroup$ So $v^{<i>}$ can be read as a word embedding vector? Isn't it calculated using $x^{<i>}$ and the weight matrix $W^{v}$, which is learned within a layer? I can see how the aggregated vector would contain the necessary information if the different $v^{<i>}$ were known outside the layer, but how would it work in this? $\endgroup$
    – Jozdien
    Jul 29 at 10:58
  • $\begingroup$ A complexity in Transformer is that there are multiple embeddings. 1. each literal word e.g "love" is converted into an integer $index$. 2. The $index$ extracts a word embedding vector $e$. 3. $e$ is added with the positional encoding into $x$. 4. $x$ and $W^v$ generate $v$. These $e$, $x$, $v$ can all be regarded as embedding. For instance, regarding the positionally encoded vector $x$, "sushi" and "pasta" in the same position in "i love sushi" and "i love pasta" will be in close proximity in it vector space $X$. $\endgroup$
    – mon
    Jul 30 at 15:33
  • $\begingroup$ If "word embedding" is meant to be in the context of word2vec in which $king - man = queen - woman$ stands, then $v$ is not such a "word embedding" vector. Such "word embedding" is $e$ which exists in a different embedding space from the one where $v$ exists. $\endgroup$
    – mon
    Jul 30 at 15:44
  • $\begingroup$ I think I understand what you mean. But as far as the layer that the summed values are passed to is concerned, it has the aggregated vector giving information about $v^{<i>}$, and has the values of $x$ for each word. But it doesn't have a mapping between them, right? As in, it doesn't have the information to understand how the different words $x$ lie in the vector space $v$. $\endgroup$
    – Jozdien
    Jul 30 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.