0
$\begingroup$

In attention, the context vector ($c$) is derived from the sum of the attention weights ($\alpha$) multiplied by the encoder hidden states ($h$), where the weights are obtained by multiplying the decoder hidden state and the encoder states.

$c_i = \Sigma_j^{T_x} \alpha_{ij} h_j$

My question is, why calculate this context vector and simply not forward the attention weights, as these could indicate how much to focus on each of the encoder states.

Could somebody explain the intuition behind this?

$\endgroup$
0
$\begingroup$

The attention weights themselves do not carry any information about the encoded sentence. It only tells you something like: the $n$-th and $m$-th word carry important information for generating the next word, but not what the words actually are. Moreover, it is a variable-length vector (of the same length as the input size), it would be hard to do anything with it in the decoder.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.