The model of LSTM with more than one unit

Question

In stacked LSTM, for example: 2 LSTM layers, LSTM_1 in order to pass the output of every time step to LSTM_2, so it needs to return hidden state value in every time step, like the architecture I drew below:

But the picture above only has one unit, so I am wondering if it has more than one unit, for example memory unit=2, what will this model look like.

or

or can someone point out the wrong part, or give a sample of visualize architecture of LSTM model with multiple units, thanks!

---

perhaps my description was not clear enough, so i try to give a more clear description and graph of the question: I try to illustrate the question as the picture below, and my question is: is the architect correct when LSTM model has 2_units,2_layers.

Answer 1

The last image that you draw, where (at a given timestep) each cell of the second layer receives input from all cells in the first layer, is the right one.

You can think of recurrent layers as fully connected layers which receive sequences of input and apply their transformations at each timestep. They just receive other inputs depending on their previous inputs but the connectivity between layer is exactly the same as a fully connected : each unit of a given layer performs a weighted average of the activations of all the units in the previous layer (or of the inputs if it's the first layer).