# Which time step output should be used in a LSTM network?

Let's take a LSTM network with one layer and two hidden units. Let's take that the number of time steps are 4, then the input x is:

\begin{align} x = \big(x\small(t),\space x\small(t-1),\space x\small(t-2),\space x\small(t-3)\big) \end{align}

And the ouput H is of the form:

\begin{align} H = \begin{bmatrix} h_{0}(t)&h_{0}(t-1)&h_{0}(t-2)&h_{0}(t-3)\\ h_{1}(t)&h_{1}(t-1)&h_{1}(t-2)&h_{1}(t-3) \end{bmatrix} \end{align}

where the rows of H represent each hidden unit output, and the columns each time step output.

My concern is how to manipulate/choose the outputs in the most reasonable way. I'm showing to possible options which are depicted in the figure below:

In Option_1, the final outputs h(t) of both hidden units are fed to the dense layer alongside with all the previous hidden states (or previous outputs). In Option_2, only the last outputs h(t) are used.

My personal considerations:

My intuition is that the obvious thing to do is going for Option_2, since h(t) depends on h(t-1), and h(t-1) on h(t-2) and so on... Then, I see at Option_1 as a waste of resources since it implies some kind of redundancy... But in Deep Learning one never knows...

Question:

Is there a common practice based on math or meaningful reasons for choosing one of both approaches? Is it only a matter of trial and error?