This blog covers the basics of LSTMs.

A forget gate is defined as :

$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t]+ b_f)$$

At this point the linear algebra confuses me more than it should. The syntax of $W\cdot [h,x]$ is confusing in this context. I think a vector should go into the activation function since the output $f$ is a vector, but the syntax of the forget gate above implies that the input has $2$ columns because $[h,x]$ will be an $n\times 2$ matrix

For the sake of example lets say ...

\begin{align} W &= \begin{bmatrix} 0 & 1 \\ 2 &3 \end{bmatrix}\\ h &= \begin{bmatrix} -1 \\ 2 \end{bmatrix}\\ x &= \begin{bmatrix} 3 \\ 0 \end{bmatrix}\\ b &= \begin{bmatrix} 1 \\ -2 \end{bmatrix}\end{align}

Can anyone give the final vector that goes into the sigmoid function ?

I think the math is

$$ \begin{bmatrix} 0 & 1 \\ 2 & 3 \end{bmatrix}\begin{bmatrix} -3 & 3 \\ 2 & 0 \end{bmatrix} + \begin{bmatrix} 1 \\ -2\end{bmatrix} = \begin{bmatrix} 2 & 0 \\ 4 & 6\end{bmatrix}+ \begin{bmatrix} 1 \\ -2\end{bmatrix} = \text{ Something wrong}$$


Note that $$[h_{t-1}, x_t]$$ is the concatenation of two vectors. In your example, it would be: $$[h_{t-1}, x_t] = [-1, 2 , 3, 0]$$ and then the dimensions of $W_f$ would be $2 \times 4$, where $2$ is the dimension of the output of the LSTM cell, i.e. the activation $h_t$, that you defined to be of dimension $2$.

Hence, $$W_f \cdot [h_{t-1}, x_t] $$ is a multiplication of a matrix of dimension $2\times4$ by a vector of $4$, which will return a vector of dimesion $2$. And then the sigmoid function will be applied point wise on each of the two elements of the result.

Hope it makes sense.

  • 1
    $\begingroup$ thanks for walking me through the math. I was not aware that $$[h_{t-1}, x_t]$$ was a concatenation operation $\endgroup$
    – sam
    Mar 11 '19 at 1:37

I interpret it as

$$f_t = \sigma \left(W_f\cdot \begin{bmatrix} h_{t-1} \\ x_t\end{bmatrix} + b_f\right)$$

That is $W_f$ has as many columns as the entries of $h_{t-1}$ and $x_t$. $W_f$ also has as many rows as $b_f$. This would make the dimension matches and prodcues a vector output.

  • $\begingroup$ this is a good point, and thank you for pointing out an equivalent representation of the math operation $\endgroup$
    – sam
    Mar 11 '19 at 1:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.