Structure of a multilayered LSTM neural network?

Question

I implemented a LSTM neural network model in Keras. However, how the codes worked under the hood was not quite clear. I want to know if it worked the way I guessed how it worked?

For example: Say there's a 2-layer LSTM network with 10 units in each layer. The inputs are a sequence data $X_1, X_2, X_3, X_4, X_5$.

So when the inputs are entered into the network, $X_1$ will be thrown into the network first and be connected to every unit in the first layer. And it will generate 10 hidden states/10 memory cell values/10 outputs. Then the 10 hidden states, 10 memory cell values and $X_2$ will be connected to the 10 units again, and generate another 10 hidden states/10 memory cell values/10 outputs and so on?

After all 5 $X_i$'s are entered into the network, the 10 outputs from $X_5$ from the first layer are then used as the inputs for the second layer. The other outputs from $X_1$ to $X_4$ are not used. And the the 10 outputs will be entered into the second layer one by one again. So the first from the 10 will be connected to every unit in the second layer and generate 10 hidden states/10 memory cell values/10 outputs. The 10 memory cell values/10 hidden states and then the second value from the 10 will be connected and so forth?

After all these are done, only the final 10 outputs from the layer 2 will be used? Is this how the LSTM network works? Thanks!

Answer 1

• For layer 1, After all 5 Xi's are entered into the network, the 10 outputs from X5 from the first layer are then used as the inputs for the second layer. The other outputs from X1 to X4 are not used.

No, the outputs from all time steps 1,2,...,5 are used as the input of the second layer.

• For layer 2, After all these are done, only the final 10 outputs from the layer 2 will be used?

It depends on you. Some people choose to use only the final size=10 output as an encoding of all the input. Some people choose to average or concat all the outputs of second layers. When using attention, all the outputs are used.

Here is a illustration of a 3-layer LSTM. Notice that all the outputs from the lower layer, not only the final output, are fed into the higher layer.

.