I am working on LSTMs and LSTM AutoEncoders, trying different types of architectures for multivariate time series data, using Keras.
Since it is not really practical to use relu in LSTM because of exploding gradients, I added a Dense layer following LSTM, so it is like:
model = Sequential() model.add(LSTM(number_of_features, batch_input_shape = (batch_size, time_steps, number_of_features), return_sequences = True)) model.add(Dense(number_of_features))
What I want to know is:
- Is this fully connected Dense layer connected to only the last step in LSTM? Or does it add a fully connected Dense layer for all time steps?
When I checked the number of parameters to be sure about this. Dense layer has number_of_features $\times$ (number_of_features + 1) parameters, which implies this Dense layer is applied to all time steps in LSTM network. This makes sense since I set return_sequences = True, but even when I set it to False, this does not change, which made me doubt my understanding.
- How does Dense work with LSTM with Return_Sequences?
- What is its different from TimeDistributed layer?
- Why changing return_sequences to False did not result in a reduction in number of parameters of Dense layer, from number_of_features $\times$ (number_of_features + 1), to (number_of_features + 1)?