I'm trying to figure out the dimensions of each variables in an RNN in the forget layer, however, I'm not sure if I'm on the right track: Colah GitHub

$x_t$: Input of size $m*1$ vector

$h_{t-1}$: Hidden state of size $n*1$ vector

And $[x_t, h_{t-1}]$ is a concatenation, if $x_t=[1, 2, 3], h_{t-1}=[4, 5, 6]$, then $[x_t, h_{t-1}]=[1, 2, 3, 4, 5, 6]$

$w_f$: weights of size $k*(m+n)$ matrix, where $k$ is the number of cell states, if $m=3$, and $n=3$ in the above example, and if we have 3 cell states, then $w_f=3*3 $ matrix.

$b_f$: bias of size $k*1$ vector, where $k$ is the number of cell states, since $k=3$ as the above example, then $b_f$ is a $3*1$ vector.

If we set $w_f$ to be: \begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 6 \\ 5 & 6 & 7 & 8 & 9 & 10 \\ 3 & 4 & 5 & 6 & 7 & 8 \\ \end{bmatrix}

And $b_f$ to be: $[1, 2, 3]$

Then $W_f . [h_{t-1}, x_t] =$

$$ \begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 6 \\ 5 & 6 & 7 & 8 & 9 & 10 \\ 3 & 4 & 5 & 6 & 7 & 8 \\ \end{bmatrix} . \begin{bmatrix} 1 \\ 2 \\ 3 \\ 4 \\ 5 \\ 6 \\ \end{bmatrix} =\begin{bmatrix} 91 & 175 & 133\end{bmatrix}$$

Then we can add the bias, $W_f . [h_{t-1}, x_t] + b_f=$

$$\begin{bmatrix} 91 & 175 & 133\end{bmatrix} + \begin{bmatrix} 1 & 2 & 3\end{bmatrix}=\begin{bmatrix} 92 & 177 & 136\end{bmatrix}$$

Then we feed them into a sigmoid function: $\frac{1}{1+e^{-x}}$, where $x=\begin{bmatrix} 92 & 177 & 136\end{bmatrix}$, hence we perform this function element wise, and get \begin{bmatrix} 1 & 1 & 1\end{bmatrix}.

Which means for each cell state, $C_{t-1}$, (there are $k=3$ cell states), we allow it to pass to the next layer.

Is the above assumption correct?

This also means that the number of cell state and hidden state is the same?

I'm trying to figure out the dimensions of each variables in an RNN in the forget layer, however, I'm not sure if I'm on the right track. The next picture and equation is from Colah's blog post "Understanding LSTM Networks":

where:

• $x_t$ is input of size $m*1$ vector
• $h_{t-1}$ is hidden state of size $n*1$ vector
• $[x_t, h_{t-1}]$ is a concatenation (for example, if $x_t=[1, 2, 3], h_{t-1}=[4, 5, 6]$, then $[x_t, h_{t-1}]=[1, 2, 3, 4, 5, 6]$)
• $w_f$ is weights of size $k*(m+n)$ matrix, where $k$ is the number of cell states (if $m=3$, and $n=3$ in the above example, and if we have 3 cell states, then $w_f=3*3 $ matrix)
• $b_f$ is bias of size $k*1$ vector, where $k$ is the number of cell states (since $k=3$ as the above example, then $b_f$ is a $3*1$ vector).

If we set $w_f$ to be: \begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 6 \\ 5 & 6 & 7 & 8 & 9 & 10 \\ 3 & 4 & 5 & 6 & 7 & 8 \\ \end{bmatrix}

And $b_f$ to be: $[1, 2, 3]$

Then $W_f . [h_{t-1}, x_t] =$

$$ \begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 6 \\ 5 & 6 & 7 & 8 & 9 & 10 \\ 3 & 4 & 5 & 6 & 7 & 8 \\ \end{bmatrix} . \begin{bmatrix} 1 \\ 2 \\ 3 \\ 4 \\ 5 \\ 6 \\ \end{bmatrix} =\begin{bmatrix} 91 & 175 & 133\end{bmatrix}$$

Then we can add the bias, $W_f . [h_{t-1}, x_t] + b_f=$

$$\begin{bmatrix} 91 & 175 & 133\end{bmatrix} + \begin{bmatrix} 1 & 2 & 3\end{bmatrix}=\begin{bmatrix} 92 & 177 & 136\end{bmatrix}$$

Then we feed them into a sigmoid function: $\frac{1}{1+e^{-x}}$, where $x=\begin{bmatrix} 92 & 177 & 136\end{bmatrix}$, hence we perform this function element wise, and get \begin{bmatrix} 1 & 1 & 1\end{bmatrix}.

Which means for each cell state, $C_{t-1}$, (there are $k=3$ cell states), we allow it to pass to the next layer.

Is the above assumption correct?

This also means that the number of cell state and hidden state is the same?