understanding linear algebra of a forget gate

Question

This blog covers the basics of LSTMs.

A forget gate is defined as :

$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t]+ b_f)$$

At this point the linear algebra confuses me more than it should. The syntax of $W\cdot [h,x]$ is confusing in this context. I think a vector should go into the activation function since the output $f$ is a vector, but the syntax of the forget gate above implies that the input has $2$ columns because $[h,x]$ will be an $n\times 2$ matrix

For the sake of example lets say ...

\begin{align} W &= \begin{bmatrix} 0 & 1 \\ 2 &3 \end{bmatrix}\\ h &= \begin{bmatrix} -1 \\ 2 \end{bmatrix}\\ x &= \begin{bmatrix} 3 \\ 0 \end{bmatrix}\\ b &= \begin{bmatrix} 1 \\ -2 \end{bmatrix}\end{align}

Can anyone give the final vector that goes into the sigmoid function ?

I think the math is

$$ \begin{bmatrix} 0 & 1 \\ 2 & 3 \end{bmatrix}\begin{bmatrix} -3 & 3 \\ 2 & 0 \end{bmatrix} + \begin{bmatrix} 1 \\ -2\end{bmatrix} = \begin{bmatrix} 2 & 0 \\ 4 & 6\end{bmatrix}+ \begin{bmatrix} 1 \\ -2\end{bmatrix} = \text{ Something wrong}$$

Answer 1

Note that $$[h_{t-1}, x_t]$$ is the concatenation of two vectors. In your example, it would be: $$[h_{t-1}, x_t] = [-1, 2 , 3, 0]$$ and then the dimensions of $W_f$ would be $2 \times 4$, where $2$ is the dimension of the output of the LSTM cell, i.e. the activation $h_t$, that you defined to be of dimension $2$.

Hence, $$W_f \cdot [h_{t-1}, x_t] $$ is a multiplication of a matrix of dimension $2\times4$ by a vector of $4$, which will return a vector of dimesion $2$. And then the sigmoid function will be applied point wise on each of the two elements of the result.

Hope it makes sense.

Answer 2

I interpret it as

$$f_t = \sigma \left(W_f\cdot \begin{bmatrix} h_{t-1} \\ x_t\end{bmatrix} + b_f\right)$$

That is $W_f$ has as many columns as the entries of $h_{t-1}$ and $x_t$. $W_f$ also has as many rows as $b_f$. This would make the dimension matches and prodcues a vector output.