Let's take a LSTM network with one layer and two hidden units. Let's take that the number of time steps are 4, then the input x is:

\begin{align} x = \big(x\small(t),\space x\small(t-1),\space x\small(t-2),\space x\small(t-3)\big) \end{align}

And the ouput H is of the form:

\begin{align} H = \begin{bmatrix} h_{0}(t)&h_{0}(t-1)&h_{0}(t-2)&h_{0}(t-3)\\ h_{1}(t)&h_{1}(t-1)&h_{1}(t-2)&h_{1}(t-3) \end{bmatrix} \end{align}

where the rows of H represent each hidden unit output, and the columns each time step output.

My concern is how to manipulate/choose the outputs in the most reasonable way. I'm showing to possible options which are depicted in the figure below:

enter image description here

In Option_1, the final outputs h(t) of both hidden units are fed to the dense layer alongside with all the previous hidden states (or previous outputs). In Option_2, only the last outputs h(t) are used.

My personal considerations:

My intuition is that the obvious thing to do is going for Option_2, since h(t) depends on h(t-1), and h(t-1) on h(t-2) and so on... Then, I see at Option_1 as a waste of resources since it implies some kind of redundancy... But in Deep Learning one never knows...


Is there a common practice based on math or meaningful reasons for choosing one of both approaches? Is it only a matter of trial and error?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.