How do attention mechanisms in RNNs learn weights for a variable length input

Question

Attention mechanisms in RNNs are reasonably common to sequence to sequence models.

I understand that the decoder learns a weight vector $\alpha$ which is applied as a weighted sum of the output vectors from the encoder network. This is used to produce a new input vector.

What I don't understand is that the learned weight vectors $\alpha$ must be a fixed size vector because it's treated as learned weights, but it's applied to a variable length sequence.

If someone could help me understand this particular mechanism I'd appreciate it.

Answer 1

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size.

For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and target sequence $\boldsymbol{y} = y_1...y_{T_y}$ (where $T_y$ can also vary from one target to another), weight $\boldsymbol{\alpha}_i = (\alpha_{i1},\cdots,\alpha_{iT_x})$ is calculated for target element $y_i$ as follows $$\alpha_{ij}=\frac{\text{exp}(e_{ij})}{\sum_{k=1}^{T_x}\text{exp}(e_{ik})}$$ where $e_{ij}$ is calculated by neural network $a$ that receives hidden state $s_{i-1}$ of decoder (decoder generates the target sequence element-by-element) and hidden state $h_j$ of encoder (encoder distills the source sequence into hidden states $h_j$, where $h_j$ is the concatenation of $j$-th hidden states of forward and backward RNNs) and outputs $$e_{ij} = a(s_{i-1}, h_j).$$ In other words, network $a$ recieves a vector of size $|s| + |h|$ and outputs a number. Because of this network, the attention matrix $\boldsymbol{\alpha}$ is free in size.

Attention weights are then used to calculate context $c_i$ for target $y_i$ as follows $$c_i = \sum_{j=1}^{T_x} \alpha_{ij}h_j,$$

and so on and so forth.

Consequently, for the next (source, target) pair with lengths ($T_{x_2}$, $T_{y_2}$), the size of $\boldsymbol{\alpha}$ would also be $T_{y_2} \times T_{x_2}$ with no problem.

Answer 2

@Esmailian's answer is right, I just want to add an intuition something to make it easier to understand.

Let $h_i$ be the hidden state of the encoder, and let $s_j$ be the hidden state of the decoder.

The problem is that we have variable number $h_i$, so the weights $\alpha$ cannot be a learnable parameter because their number is changing with the input sequence length.

As a solution, we can make a factory that learns to generates alphas, the factory takes $h_i$ and $s_j$ and it spits out $\alpha_{ij}$. The factory is just a vector $v$ and a matrix $W$ that transform the concatenated vector

\begin{bmatrix} h_i \\ s_j\\ \end{bmatrix}

vector $v$ and a matrix $W$ are learnable. After all $\alpha_{ij}$ are computed, they are normalized by a softmax function, so we need them to sum up to zero ($alphas$ just scale the vectors $h_i$)