# Multi-output, multi-timestep sequence prediction with Keras

I've been searching for about three hours and I can't find an answer to a very simple question.

I have a time series prediction problem. I am trying to use a Keras LSTM model (with a Dense at the end) to predict multiple outputs over multiple timesteps using multiple inputs and a moving window. I want to do sequence-to-sequence prediction, where my model is trained on the output of every timestep, not just the last one.

What shape should my targets be? My inputs are an array of shape (number_of_moving_windows, input_window_length, number_of_features). Ought my outputs to be (number_of_moving_windows, output_window_length, number_of_series_to_predict)? Or maybe (number_of_moving_windows, output_window_length*number_of_series_to_predict)? Or what?

Aurelien Geron's textbook "Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow (2nd edition)" gives the following code for creating a single-output, 10-timestep, sequence-to-sequence target array:

# series is a (batch_size, time_steps, 1) NumPy array of random time series
# where batch_size=10000 and time_steps=n_steps+10
Y = np.empty((10000, n_steps, 10)) # each target is a sequence of 10D vectors
for step_ahead in range(1, 10 + 1):
Y[:, :, step_ahead - 1] = series[:, step_ahead:step_ahead + n_steps, 0] # This zero will drop a dimension
Y_train = Y[:7000]
Y_valid = Y[7000:9000]
Y_test = Y[9000:]


How do I change this to have a 10-timestep, 3-output target? Ought my target shape to be (batch_size, 10, 3) or (batch_size, 30) or what?

Also, do I make the final Dense layer in my network a Dense(30)?

EDIT:

For an example, suppose my data are this dataframe:

import pandas as pd
import numpy as np

dummy_data = np.concatenate([np.arange(100, 113).reshape(-1, 1),
np.arange(200, 213).reshape(-1, 1),
np.arange(300, 313).reshape(-1, 1)],
axis=1)

dummy_data = pd.DataFrame(dummy_data, columns=["A", "B", "C"])

A       B       C
0   100     200     300
1   101     201     301
2   102     202     302
3   103     203     303
4   104     204     304
5   105     205     305
6   106     206     306
7   107     207     307
8   108     208     308
9   109     209     309
10  110     210     310
11  111     211     311
12  112     212     312


I want to predict, for each window of four input timesteps (t-3, t-2, t-1, t), for all of A, B and C together, the values at the following three timesteps (t+1, t+2, t+3). So input_window_length is 4, number_of_series_to_predict equals number_of_features equals 3, and output_window_length is 3. These imply that number_of_moving_windows is 4.

Then my windowed inputs for my training set are:

np.array([[[100, 200, 300],
[101, 201, 301],
[102, 202, 302],
[103, 203, 303]],

[[101, 201, 301],
[102, 202, 302],
[103, 203, 303],
[104, 204, 304]],

[[102, 202, 302],
[103, 203, 303],
[104, 204, 304],
[105, 205, 305]],

[[103, 203, 303],
[104, 204, 304],
[105, 205, 305],
[106, 206, 306]]])


I am asking about the shape of the corresponding targets. Do I create an array of shape (number_of_moving_windows, output_window_length*number_of_series_to_predict), i.e. (4, 9)? Like this one:

np.array([[104, 204, 304, 105, 205, 305, 106, 206, 306],
[105, 205, 305, 106, 206, 306, 107, 207, 307],
[106, 206, 306, 107, 207, 307, 108, 208, 308],
[107, 207, 307, 108, 208, 308, 109, 209, 309]])


Or do I make it (number_of_moving_windows, output_window_length, number_of_series_to_predict), i.e. (4, 3, 3)? Like this one:

np.array([[[104, 105, 106],
[105, 106, 107],
[106, 107, 108],
[107, 108, 109]],

[[204, 205, 206],
[205, 206, 207],
[206, 207, 208],
[207, 208, 209]],

[[304, 305, 306],
[305, 306, 307],
[306, 307, 308],
[307, 308, 309]]])


Or (number_of_moving_windows, number_of_series_to_predict, output_window_length), i.e. (4, 3, 3) again, but with the last two dimension swapped? Like this one:

np.array([[[104, 204, 304],
[105, 205, 305],
[106, 206, 306],
[107, 207, 307]],

[[105, 205, 305],
[106, 206, 306],
[107, 207, 307],
[108, 208, 308]],

[[106, 206, 306],
[107, 207, 307],
[108, 208, 308],
[109, 209, 309]]])


• To get an output on every step, you have to make return_sequence=True for all LSTM layers
• You will get 3 output for every step which make 30, please help me understand the confusion. To make this seq. you can wrap inside the TimeDistributed layer though it is not needed if it's the last layer. Commented May 11, 2021 at 14:21