Multi-Step Forecast for Multivariate Time Series (LSTM) Keras

Question

I have been trying to understand how to build LSTM model for multivariate time series forecast using Keras but I am still unsure how to present the data in the correct shape.

My Dataset:
•   5 cities.
•   Each with 2 features. Temperature and humidity.
•   Daily records of the last 10 weeks(Monday, Tuesday, …, Sunday)

What I want to do:

Given Monday’s record of the new week of a city, I'd like to forecast the Temperature and humidity for the remaining 6 days of that city. I.e. Multi-Step Forecast for Multivariate Time Series. Output shape(6,2)

How I have started off:

For each city, the input shape [(num_samples, num_time_steps, num_features) ] would be (10, 7, 2).

For 10 weeks, I will have five unique samples(5-cities) with the same shape (70, 2). So if I stack all vertically I will have (350, 2) or 3D shape (50,7,2). Then create a supervised series with lag 1, I will have a shape(244, 4)

# Split train/test data. 
train on 7-weeks. So input_shape= 5*(7,7,2) = (35,7,2)
test on 3-weeks. . So input_shape= 5*(3,7,2) = (15,7, 2)

What I am confused about:

The above layout seems to disregard the unique nature of each sample. I looked at this but still, a bit confused on how to transform it to a regression model.

I want the network to train each city's data separately as in this pic I would appreciate any suggestion. Thanks

Answer 1

You should use Seq2Seq models. Seq2seq models represent, in the RNN family, the best for multistep predictions. More classical RNNs, on the other side, are not that good for predicting long sequences.

If you need to implement a seq2seq model in TensorFlow 2.0 / Keras, each model follows the following structure:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, RepeatVector, Dense, TimeDistributed
from tensorflow.keras.activations import elu

# these are just made up hyperparameters, change them as you wish
hidden_size = 50


seq2seq = Sequential([

    LSTM(hidden_size, input_shape = (input_sequence_length, no_vars)),

    RepeatVector(prediction_length), 

    LSTM(hidden_size, return_sequences = True), 

    Dense(hidden_size, activation = elu)

    TimeDistributed(Dense(1, activation = elu))

])

and then train it as usual with seq2seq.compile() and seq2seq.fit().

If you want to stack more LSTM() layers on top of each other, simply add them to the model I depicted above. Please keep in mind that LSTM models are very computationally expensice; without a good GPU even this "basic" model could be painfully long to train.

One modification I'd suggest, looking at your image, is to make the LSTM-encoder and -decoder parts of equal size and depth.

Alternatively, you can implement a more classical "Autoencoder-like" architecture, with LSTM() layers for encoding and decoding, and Dense() layers in the middle. However, seq2seq models are the most powerful at the moment.

To my knowledge, the only models more state-of-the-art than this are attention models. The problem is that they are so much state-of-the-art that TensorFlow/Keras doesn't have built-in layers for them, and you'd have to create your own custom layers (it's a pain). The only extensive implementation of attention models I found is from this blog post, but things are going to be very very complicated here. I didn't try to implement one yet.