1
$\begingroup$

So I was reading about the LSTM architecture and I was having trouble understanding a certain aspect of it. This article mentions the step in question near the bottom of the page. Here is the image given:enter image description here

The issue I have is this: If ot is the result of an operation on the concatenation of ht-1 and xt, then ot will be larger than ht-1. Then, ht is calculated by an operation on the now larger ot. ht is then 'passed' on to the next timestep. So wouldn't the size of h continually increase over time?

I'm definitely misunderstanding some part of this, so thanks in advance for any help. :)

$\endgroup$
1
$\begingroup$

o(t) is not the result of concatenation of h(t-1) and x(t), but a simple matrix multiplication. See wikipedia for further details:

https://en.wikipedia.org/wiki/Long_short-term_memory

LSTM operation

$\endgroup$
0
$\begingroup$

Keep in mind how matrix multiplication works with regards to the dimensions:

Multiplying a matrix with dimensions $n,m$ by a matrix with dimensions $m,k$ results in a matrix of size $n,k$. Therefore, you can add as many rows as you like to the second matrix with no change to the shape of the result of the matrix multiplication. But of course the first matrix needs to be compatible with this, i.e. have the same number of columns as the second matrix has rows.

And that is exactly what is happening here: $$o_t = \sigma (W_o \cdot [h_{t-1},x_t]+b_o) $$

The dimensions of $W_o \cdot [h_{t-1},x_t]$ will not increase as $W_o$ has a constant number of rows $n$ and $[h_{t-1},x_t]$ has a constant number of columns $k$. Therefore, the resulting matrix will always be of size $n,k$.

An alternative way to write the above equation (e.g. see here and the post by @cho_uc) is: $$o_t^{alt} = \sigma (W_o^{alt} \cdot h_{t-1} + U_o \cdot x_t+b_o) $$

Here the concatenation is simply split into two separate terms ($W_o^{alt} \cdot h_{t-1}$ and $U_o \cdot x_t$) which are added. While in your notation $W_o$ contains the weights of $W_o^{alt}$ and $U_o$. But the result is the same.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.