I am trying to learn LSTM's, and struggling a bit with the structure and the inputs/outputs of LSTM layers. Say I have a network definition like this:
timesteps = 50
features = 10
inp = Input(shape=(timesteps, features,), name="input_layer")
x = LSTM(128, return_sequences=True, name="lstm1")(inp)
x = LSTM(64, return_sequences=False, name="lstm2")(x)
x = Dense(128, activation="relu", name="dense1")(x)
x = Dense(64, activation="relu", name="dense2")(x)
x = Dense(1, activation='sigmoid', name="output")(x)
model = Model(inputs=inp, outputs=x)
model.compile(loss='binary_crossentropy', optimizer='adam')
I can find plenty of resources on the internet on the internal structure of the LSTM cell. For example, this has the figure with one input and some outputs. But I would first like to understand at a higher level, what inputs and outputs an LSTM layer gives in different configurations.
Lets say I have input data shape of (1000,50,10). So 1000 observations, each with 50 timesteps for 10 features.
Questions:
My understand is, the first layer ("lstm1") would take the input as shape (50, 10). One at a time for each of the 1000 observations. How are the 10 features input into the LSTM layer? Is it similar to a Dense layer, so each input feature goes into each LSTM cell with some weight? This answer seems to indicate so, but I do not quite understand what are the multiple inputs for each LSTM "unit" in that answer (perhaps even what "unit" is).
And the internal workings. Does "lstm1" here process all the 50 timesteps at once and then produce an output? After it is done for all 50 timesteps the next layer gets its input from this output? And what is the shape of that output? Is it shape=(50, 128)? So is the set of output "features" in this case the number of units in the LSTM layer? And because of "return_sequences=True", it also outputs the intermediate value at each timestep to produce the shape=(50,XXX) outputs?
This leads to my question on the "lstm2" layer. Using "return_sequences=False", does it now output a single array of 64 values? So shape=(64)? No value for every timestep but rather only one per unit at the end of the "unrolled" cell loop?