Trying to understand encoder-decoder sequential models in Keras?

Question

My understanding is that for some types of seq2seq models, you train an encoder and a decoder, and then you set aside the encoder and use only the decoder for the prediction step. For example this seq2seq time series prediction model from Uber:

Now I am trying to implement a to version of this in Keras.

This is the Keras code for a vanilla LSTM:

# define model
model = Sequential()
model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
# train model
model.fit(X, y, epochs=200, verbose=0)
# predict
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)

This is the Keras code for a stacked LSTM model:

# define model
model = Sequential()
model.add(LSTM(50, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))
model.add(LSTM(50, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
# train model
model.fit(X, y, epochs=200, verbose=0)
# predict
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

And this is the Keras code for an encoder-decoder model:

# define model
model = Sequential()
model.add(LSTM(100, activation='relu', input_shape=(n_steps_in, n_features)))
model.add(RepeatVector(n_steps_out))
model.add(LSTM(100, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(1)))
model.compile(optimizer='adam', loss='mse')
# train model
model.fit(X, y, epochs=100, verbose=0)
# predict
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)

The problem is, I don't see much difference between the encoder-decoder code and the vanilla and stacked LSTM. In particular, I don't see how we are using only the decoder in the predict step, and which variable or method in Keras corresponds to the embedding that we would be using as an input for predicting new time series?

How can I implement the code for a model similar to the one in the illustration using Keras?

Answer 1

First of all, I personally think that the code you provided is unnecessary since you are looking for architectural comparison between models. It would have been easier (and less bulky) to include the graphs of your vanilla and stacked LSTM networks for simplicity. You can do that by keras.utils.plot_model() as explained here.

Answering your questions:

I don't see how we are using only the decoder in the predict step...

Uber's model does not use the decoder component of the autoencoder. It uses a separate sequential LSTM model, judging by the graph provided.

...which variable or method in Keras corresponds to the embedding that we would be using as an input for predicting new time series?

The input to the forecasting model is simply the latent space of the autoencoder. In particular, the latent space constitutes a condensed representation of the past observations, so that you don't need to include the complete timeseries because it might be very long and include redundant information, repetition, noise, etc. Instead, you compress it into something similar to a "zip" file, that has smaller size but still includes all the essential information. Then, you train the "forecasting" network to predict future values based on the "zipped" past information instead of training it on the original timeseries.

How can I implement the code for a model similar to the one in the illustration using Keras?

Check the 4th model of this autoencoder example. First, train the autoencoder. Only after it is properly trained and optimized, train a simple LSTM forecasting model, like the one in the code you provided.

The key is to use the latent space of the autoencoder to train the forecasting model, which in the current example is the output of the 1st LSTM layer. To do that, you need to first encode your complete dataset into a latent representation using the already trained encoder, and then use these encodings to train the forecasting model in the second step.

Answer 2

It depends on what you want to achieve. The biggest difference is that in the autoencoder (encoder-decoder) the output from the hidden space ("middle LSTM") should often be used somehow, and the hidden space is often much lower (or higher) in dimensionality than the input and output.

This way you are first "encoding" the information down to very low amount of data, hoping it contains only the essential information (the gist) you are looking for.

I would start by learning about Autoencoders for instance at goodfellows deeplearningbook.org or even at wikipedia.

Your implementation above seems correct, however, the important thing is to know why you want to use an auto encoder? Is it to denoise the data or maybe create some sort of compression algorithm? Maybe to do arithmetic in the hidden space?