Understanding LSTM structure

Question

I am trying to learn LSTM's, and struggling a bit with the structure and the inputs/outputs of LSTM layers. Say I have a network definition like this:

timesteps = 50
features = 10
inp = Input(shape=(timesteps, features,), name="input_layer")
x = LSTM(128, return_sequences=True, name="lstm1")(inp)
x = LSTM(64, return_sequences=False, name="lstm2")(x)
x = Dense(128, activation="relu", name="dense1")(x)
x = Dense(64, activation="relu", name="dense2")(x)
x = Dense(1, activation='sigmoid', name="output")(x)
model = Model(inputs=inp, outputs=x)
model.compile(loss='binary_crossentropy', optimizer='adam')

I can find plenty of resources on the internet on the internal structure of the LSTM cell. For example, this has the figure with one input and some outputs. But I would first like to understand at a higher level, what inputs and outputs an LSTM layer gives in different configurations.

Lets say I have input data shape of (1000,50,10). So 1000 observations, each with 50 timesteps for 10 features.

Questions:

My understand is, the first layer ("lstm1") would take the input as shape (50, 10). One at a time for each of the 1000 observations. How are the 10 features input into the LSTM layer? Is it similar to a Dense layer, so each input feature goes into each LSTM cell with some weight? This answer seems to indicate so, but I do not quite understand what are the multiple inputs for each LSTM "unit" in that answer (perhaps even what "unit" is).

And the internal workings. Does "lstm1" here process all the 50 timesteps at once and then produce an output? After it is done for all 50 timesteps the next layer gets its input from this output? And what is the shape of that output? Is it shape=(50, 128)? So is the set of output "features" in this case the number of units in the LSTM layer? And because of "return_sequences=True", it also outputs the intermediate value at each timestep to produce the shape=(50,XXX) outputs?

This leads to my question on the "lstm2" layer. Using "return_sequences=False", does it now output a single array of 64 values? So shape=(64)? No value for every timestep but rather only one per unit at the end of the "unrolled" cell loop?

Answer 1

Adding

model.summary()

Produces this output

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_layer (InputLayer)     (None, 50, 10)            0         
_________________________________________________________________
lstm1 (LSTM)                 (None, 50, 128)           71168     
_________________________________________________________________
lstm2 (LSTM)                 (None, 64)                49408     
_________________________________________________________________
dense1 (Dense)               (None, 128)               8320      
_________________________________________________________________
dense2 (Dense)               (None, 64)                8256      
_________________________________________________________________
output (Dense)               (None, 1)                 65        
=================================================================
Total params: 137,217
Trainable params: 137,217
Non-trainable params: 0
_________________________________________________________________

1. LSTM Unit indicates dimension of Layer's output.
2. LSTM Layer (lstm1 for example) , processes 1 input (50,10 in this example) and generates 128 bit representation of each timestep.
3. lstm2 does generate a single vector with 64 values. It needs to learn ~49k parameters that are required to transform output to a single vector

References :

https://www.researchgate.net/publication/13853244

https://www.quora.com/What-is-the-meaning-of-%E2%80%9CThe-number-of-units-in-the-LSTM-cell

https://www.quora.com/What-the-difference-between-an-LSTM-memory-cell-and-an-LSTM-layer