3
$\begingroup$

I've been searching for about three hours and I can't find an answer to a very simple question.

I have a time series prediction problem. I am trying to use a Keras LSTM model (with a Dense at the end) to predict multiple outputs over multiple timesteps using multiple inputs and a moving window. I want to do sequence-to-sequence prediction, where my model is trained on the output of every timestep, not just the last one.

What shape should my targets be? My inputs are an array of shape (number_of_moving_windows, input_window_length, number_of_features). Ought my outputs to be (number_of_moving_windows, output_window_length, number_of_series_to_predict)? Or maybe (number_of_moving_windows, output_window_length*number_of_series_to_predict)? Or what?

Aurelien Geron's textbook "Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow (2nd edition)" gives the following code for creating a single-output, 10-timestep, sequence-to-sequence target array:

# series is a (batch_size, time_steps, 1) NumPy array of random time series 
# where batch_size=10000 and time_steps=n_steps+10
Y = np.empty((10000, n_steps, 10)) # each target is a sequence of 10D vectors 
for step_ahead in range(1, 10 + 1): 
    Y[:, :, step_ahead - 1] = series[:, step_ahead:step_ahead + n_steps, 0] # This zero will drop a dimension
Y_train = Y[:7000] 
Y_valid = Y[7000:9000] 
Y_test = Y[9000:]

How do I change this to have a 10-timestep, 3-output target? Ought my target shape to be (batch_size, 10, 3) or (batch_size, 30) or what?

Also, do I make the final Dense layer in my network a Dense(30)?

EDIT:

For an example, suppose my data are this dataframe:

import pandas as pd
import numpy as np

dummy_data = np.concatenate([np.arange(100, 113).reshape(-1, 1), 
                        np.arange(200, 213).reshape(-1, 1), 
                        np.arange(300, 313).reshape(-1, 1)], 
                       axis=1)

dummy_data = pd.DataFrame(dummy_data, columns=["A", "B", "C"])

     A       B       C
0   100     200     300
1   101     201     301
2   102     202     302
3   103     203     303
4   104     204     304
5   105     205     305
6   106     206     306
7   107     207     307
8   108     208     308
9   109     209     309
10  110     210     310
11  111     211     311
12  112     212     312

I want to predict, for each window of four input timesteps (t-3, t-2, t-1, t), for all of A, B and C together, the values at the following three timesteps (t+1, t+2, t+3). So input_window_length is 4, number_of_series_to_predict equals number_of_features equals 3, and output_window_length is 3. These imply that number_of_moving_windows is 4.

Then my windowed inputs for my training set are:

np.array([[[100, 200, 300],
         [101, 201, 301],
         [102, 202, 302],
         [103, 203, 303]],
 
        [[101, 201, 301],
         [102, 202, 302],
         [103, 203, 303],
         [104, 204, 304]],
 
        [[102, 202, 302],
         [103, 203, 303],
         [104, 204, 304],
         [105, 205, 305]],
 
        [[103, 203, 303],
         [104, 204, 304],
         [105, 205, 305],
         [106, 206, 306]]])

I am asking about the shape of the corresponding targets. Do I create an array of shape (number_of_moving_windows, output_window_length*number_of_series_to_predict), i.e. (4, 9)? Like this one:

np.array([[104, 204, 304, 105, 205, 305, 106, 206, 306],
        [105, 205, 305, 106, 206, 306, 107, 207, 307],
        [106, 206, 306, 107, 207, 307, 108, 208, 308],
        [107, 207, 307, 108, 208, 308, 109, 209, 309]])

Or do I make it (number_of_moving_windows, output_window_length, number_of_series_to_predict), i.e. (4, 3, 3)? Like this one:

np.array([[[104, 105, 106],
        [105, 106, 107],
        [106, 107, 108],
        [107, 108, 109]],

       [[204, 205, 206],
        [205, 206, 207],
        [206, 207, 208],
        [207, 208, 209]],

       [[304, 305, 306],
        [305, 306, 307],
        [306, 307, 308],
        [307, 308, 309]]])

Or (number_of_moving_windows, number_of_series_to_predict, output_window_length), i.e. (4, 3, 3) again, but with the last two dimension swapped? Like this one:

np.array([[[104, 204, 304],
        [105, 205, 305],
        [106, 206, 306],
        [107, 207, 307]],

       [[105, 205, 305],
        [106, 206, 306],
        [107, 207, 307],
        [108, 208, 308]],

       [[106, 206, 306],
        [107, 207, 307],
        [108, 208, 308],
        [109, 209, 309]]])

Thank you for your help.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$
  • To get an output on every step, you have to make return_sequence=True for all LSTM layers
  • The last Dense layer should reflect the output size i.e. 3 here
  • Hence, the shape of Y = (total instances, n_steps, output size)
Improve this answer
$\endgroup$
9
  • $\begingroup$ Thank you. I need my Dense layer to give an output for each of three series for each of ten periods. If I were forecasting only one period, I'd need a Dense(3). But I'm forecasting ten periods! $\endgroup$
    – MGreen
    Commented May 11, 2021 at 9:51
  • $\begingroup$ Yeah, that's why I added point #1 e.g. for [1,2,3] the y_true will be [4,5,6]. $\endgroup$
    – 10xAI
    Commented May 11, 2021 at 10:32
  • $\begingroup$ The final Dense doesn't return sequences, though, so if I use a Dense(3) I'll get three outputs, not 30. Aurelien Geron says I need one neuron per timestep in my final Dense when predicting one series, but he doesn't explain how to structure the targets when I scale that up to a Dense(30) for predicting three series for ten forecast timesteps. $\endgroup$
    – MGreen
    Commented May 11, 2021 at 12:49
  • $\begingroup$ You will get 3 output for every step which make 30, please help me understand the confusion. To make this seq. you can wrap inside the TimeDistributed layer though it is not needed if it's the last layer. $\endgroup$
    – 10xAI
    Commented May 11, 2021 at 14:21
  • $\begingroup$ The LSTM layers will return sequences, but the final Dense won't. But that's not the main problem. The main problem is that even if I have thirty numbers coming out of my network, I don't know what shape my targets should be. If I'm predicting 10 future timesteps for 1 series, Geron says I need a Dense(10) and my targets should be (batch_size, n_steps, 10). Predicting 10 future timesteps for 3 series can't possibly require a smaller Dense layer and lower-dimensional targets. So is it (batch_size, n_steps, 30) or (batch_size, n_steps, 10, 3) or (batch_size, n_steps, 3, 10) or what? $\endgroup$
    – MGreen
    Commented May 11, 2021 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.