# understanding linear algebra of a forget gate

This blog covers the basics of LSTMs.

A forget gate is defined as :

$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t]+ b_f)$$

At this point the linear algebra confuses me more than it should. The syntax of $$W\cdot [h,x]$$ is confusing in this context. I think a vector should go into the activation function since the output $$f$$ is a vector, but the syntax of the forget gate above implies that the input has $$2$$ columns because $$[h,x]$$ will be an $$n\times 2$$ matrix

For the sake of example lets say ...

\begin{align} W &= \begin{bmatrix} 0 & 1 \\ 2 &3 \end{bmatrix}\\ h &= \begin{bmatrix} -1 \\ 2 \end{bmatrix}\\ x &= \begin{bmatrix} 3 \\ 0 \end{bmatrix}\\ b &= \begin{bmatrix} 1 \\ -2 \end{bmatrix}\end{align}

Can anyone give the final vector that goes into the sigmoid function ?

I think the math is

$$\begin{bmatrix} 0 & 1 \\ 2 & 3 \end{bmatrix}\begin{bmatrix} -3 & 3 \\ 2 & 0 \end{bmatrix} + \begin{bmatrix} 1 \\ -2\end{bmatrix} = \begin{bmatrix} 2 & 0 \\ 4 & 6\end{bmatrix}+ \begin{bmatrix} 1 \\ -2\end{bmatrix} = \text{ Something wrong}$$

Note that $$[h_{t-1}, x_t]$$ is the concatenation of two vectors. In your example, it would be: $$[h_{t-1}, x_t] = [-1, 2 , 3, 0]$$ and then the dimensions of $$W_f$$ would be $$2 \times 4$$, where $$2$$ is the dimension of the output of the LSTM cell, i.e. the activation $$h_t$$, that you defined to be of dimension $$2$$.

Hence, $$W_f \cdot [h_{t-1}, x_t]$$ is a multiplication of a matrix of dimension $$2\times4$$ by a vector of $$4$$, which will return a vector of dimesion $$2$$. And then the sigmoid function will be applied point wise on each of the two elements of the result.

Hope it makes sense.

• thanks for walking me through the math. I was not aware that $$[h_{t-1}, x_t]$$ was a concatenation operation – sam Mar 11 '19 at 1:37

I interpret it as

$$f_t = \sigma \left(W_f\cdot \begin{bmatrix} h_{t-1} \\ x_t\end{bmatrix} + b_f\right)$$

That is $$W_f$$ has as many columns as the entries of $$h_{t-1}$$ and $$x_t$$. $$W_f$$ also has as many rows as $$b_f$$. This would make the dimension matches and prodcues a vector output.

• this is a good point, and thank you for pointing out an equivalent representation of the math operation – sam Mar 11 '19 at 1:40