Attention mechanisms in RNNs are reasonably common to sequence to sequence models.

I understand that the decoder learns a weight vector $\alpha$ which is applied as a weighted sum of the output vectors from the encoder network. This is used to produce a new input vector.

What I don't understand is that the learned weight vectors $\alpha$ must be a fixed size vector because it's treated as learned weights, but it's applied to a variable length sequence.

If someone could help me understand this particular mechanism I'd appreciate it.

  • 1
    $\begingroup$ bump because this link still doesn't explain how number of $\alpha$ weights can vary. Just as OP I see a big limitation - number of $\alpha$ weights indeed has to be constrained, meaning LSTM must produce fixed number of encoded timesteps, thus we loose the benefit of LSTM. An answer would be greatly appriciated $\endgroup$
    – Kari
    Commented Mar 9, 2018 at 5:16

2 Answers 2


Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size.

For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and target sequence $\boldsymbol{y} = y_1...y_{T_y}$ (where $T_y$ can also vary from one target to another), weight $\boldsymbol{\alpha}_i = (\alpha_{i1},\cdots,\alpha_{iT_x})$ is calculated for target element $y_i$ as follows $$\alpha_{ij}=\frac{\text{exp}(e_{ij})}{\sum_{k=1}^{T_x}\text{exp}(e_{ik})}$$ where $e_{ij}$ is calculated by neural network $a$ that receives hidden state $s_{i-1}$ of decoder (decoder generates the target sequence element-by-element) and hidden state $h_j$ of encoder (encoder distills the source sequence into hidden states $h_j$, where $h_j$ is the concatenation of $j$-th hidden states of forward and backward RNNs) and outputs $$e_{ij} = a(s_{i-1}, h_j).$$ In other words, network $a$ recieves a vector of size $|s| + |h|$ and outputs a number. Because of this network, the attention matrix $\boldsymbol{\alpha}$ is free in size.

Attention weights are then used to calculate context $c_i$ for target $y_i$ as follows $$c_i = \sum_{j=1}^{T_x} \alpha_{ij}h_j,$$

and so on and so forth.

Consequently, for the next (source, target) pair with lengths ($T_{x_2}$, $T_{y_2}$), the size of $\boldsymbol{\alpha}$ would also be $T_{y_2} \times T_{x_2}$ with no problem.


@Esmailian's answer is right, I just want to add an intuition something to make it easier to understand.

Let $h_i$ be the hidden state of the encoder, and let $s_j$ be the hidden state of the decoder.

The problem is that we have variable number $h_i$, so the weights $\alpha$ cannot be a learnable parameter because their number is changing with the input sequence length.

As a solution, we can make a factory that learns to generates alphas, the factory takes $h_i$ and $s_j$ and it spits out $\alpha_{ij}$. The factory is just a vector $v$ and a matrix $W$ that transform the concatenated vector

\begin{bmatrix} h_i \\ s_j\\ \end{bmatrix}

vector $v$ and a matrix $W$ are learnable. After all $\alpha_{ij}$ are computed, they are normalized by a softmax function, so we need them to sum up to zero ($alphas$ just scale the vectors $h_i$)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.