Multivariate time series forecasting with LSTM

Question

I have the following problem:

$\mathbf{Y}(t)$ = real valued random vector of observations at time t, $Y_i(t) \in R_{(0, 1)}$

$\mathbf{X}(t)$ = real valued random vector of observations at time t, $X_i(t) \in R$

I want to forecast $\mathbf{Y}(t+1)$, given $\mathbf{X}(t+1)$ and $\mathbf{Y}$ history.

To sum up I'd like to forecast a vector of observations given its history and another vector of observations as the same timestamp.

To make it clearer: let's say that I want to forecast daily temperatures in London and Dublin, knowing the same day temperatures in Manchester and Liverpool and London and Dublin (and Manchester and Liverpool) historical data. It's a bit like a kalman filter where you want to estimate the state variables given some output.

How would you set up this problem in an LSTM?

Answer 1

1)First and most important, do not give Yi(t) history as feature. You will just end in a model that replicates the previous input to minimize the error, a cheating model. For more detail, you can have a look at my explanations at two different questions:

https://stackoverflow.com/questions/52778922/stock-prediction-gru-model-predicting-same-given-values-instead-of-future-stoc/52786399?noredirect=1#comment93125020_52786399

https://stackoverflow.com/questions/52252442/how-to-handle-shift-in-forecasted-value/53141558#53141558

2)Create your labels by sliding your Y(t) one step forward so that your each sample will have a label of Y(t+1). That means you will delete sample #1 as a result.

3)Use a time-window for each sample for your features. Do not provide just your features as x(t) for the label y(t+1). For example, with a window size w = 10, provide x(t-9),x(t-8),.....,x(t-1),x(t+1) as a single sample for label y(t+1). Then you will be boosting the sequential nature of LSTM, possibly acquiring greater performance.

4)You can use Keras for your LSTM regression task, have a look at simple code piece from my old works, I modified it for your task:

nn = Sequential()
nn.add(LSTM(80, batch_input_shape=(64,11,20), return_sequences=True, recurrent_dropout = 0.1))
nn.add(LSTM(60, recurrent_dropout = 0.2))
nn.add(Dropout(0.2))
nn.add(Dense(40, activation='relu'))
nn.add(Dropout(0.2))
nn.add(Dense(10, activation='relu'))
nn.add(Dense(1))
nn.compile(loss= 'mse, optimizer= 'adam')
nn.fit(x_train, y_train, epochs=15, batch_size=64, shuffle=True, validation_data=(x_dev, y_dev))
y_pred = nn.predict(x_dev, batch_size=64)

where batch_input_shape=(64,10,25) explains that your batch size is 64 (trains 64 samples per gradient descent), your window size is 10, and you have 25 different features. You can delete anything about dropout in the code if you get confused; they are just for preventing overfitting.

Note: Do not forget to normalize your numeric input data at the start!

Hope I could help. Good Luck!