keras CNN lstm add model depth

Question

Would anyone have any advice on how to add model depth? This works below but I was hoping to experiment with adding in additional non-TimeDistributed layers.

# reshape from [samples, timesteps] into [samples, subsequences, timesteps, features]
n_features = 1
n_seq = 2
n_steps = 2


# define model
model = Sequential()
model.add(TimeDistributed(Conv1D(filters=16, kernel_size=1, activation='relu'), input_shape=(None, n_steps, n_features)))
model.add(TimeDistributed(MaxPooling1D(pool_size=2)))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(50, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
# fit model
model.fit(X, y, epochs=500, verbose=0)

For example, if I add this in:

# define model
model = Sequential()
model.add(TimeDistributed(Conv1D(filters=16, kernel_size=1, activation='relu'), input_shape=(None, n_steps, n_features)))
model.add(TimeDistributed(MaxPooling1D(pool_size=2)))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(50, activation='relu'))
model.add(LSTM(40, activation='relu'))
model.add(LSTM(30, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
# fit model
model.fit(X, y, epochs=500, verbose=0)

This will throw ValueError: Input 0 is incompatible with layer lstm_2: expected ndim=3, found ndim=2

Any tips help! Sorry not a lot of Wisdom here but learning :)

Answer 1

CTry to change

model.add(LSTM(50, activation='relu'))

to

model.add(LSTM(50, activation='relu', return_sequences=True))

as explained in the documentation ( https://keras.io/layers/recurrent/#lstm ) return_sequences allow the LSTM to return the complete sequence of vectors of the LSTM rather than a single vector.