5
$\begingroup$

I have built a generic stacked lstm model of the form:

self.model = tf.keras.Sequential([
            lstm_layer(n_neurons, return_sequences=True),
            lstm_layer(n_neurons, return_sequences=True),
            lstm_layer(n_neurons, return_sequences=True),
            lstm_layer(n_neurons),
            tf.keras.layers.Dense(n_output)
        ])

with the data shaped as x_train = (total_samples,timestep, features) and y_test = (total_samples,n_output,1) and then I fit the model with a validation split of 30%.

My question is:

Is there a good theoretical reason for why the model perfectly predicts one step ahead (n_output = 1) but when I try to set n_output>1 the model becomes very, very imprecise? (loss<val_loss from epoch 1) Is it a model/data dependant thing or am I botching the data preparation?

$\endgroup$
3
  • $\begingroup$ Have you tried some hyperparametrization? Is that the one the only architecture you tried? Maybe (without knowing the data prep you carried out) one reason which could be ofertting to only next values ahead... $\endgroup$
    – German C M
    Jan 29, 2021 at 6:46
  • $\begingroup$ I have also tried adding convolutional layers and an encoder-decoder. Differenr time steps and batch sizes but the same issue spans all these models. The data is orepared very simply as is described in the question, just stacks of points $\endgroup$
    – Fra
    Jan 29, 2021 at 9:16
  • $\begingroup$ What is the value of n_neurons, timestep, n_output. Also, what is your y_train, let's say for 1st batch e.g. [1,2,3,4,5,6,7,8,9,10] $\endgroup$
    – 10xAI
    Feb 2, 2021 at 10:03

2 Answers 2

1
+50
$\begingroup$

Welcome to the community Fra, below you can find a worked out example implementing a multivariate several input features (as I think is your case) time series forecasting, predicting multiple future steps (multi-step forecast), applying bayesian hyperparametrization.

It is based on a from simpler to more complex approach, so you can see there are few layers in the architecture to begin with, and what is being hyperparametrized is the units in such layers. This is a whole worked out example, in case you can almost directly apply it and try out modelling first with a single output node for a single-step prediction, and then add more future step predictions to check if you still run on the same issue as you described (beggining with n_output = 1 and then incrementing this, e.g. in my case it is 72 future values to predict).

For a dataset of this type:

enter image description here

and trying to predict several future time steps (the deep blue dots) for the temperature values: enter image description here

we have some code like:

  • train-validation dataset splitter functions:

      def multivariate_data(dataset, target, start_index, end_index, history_size, target_size, step, single_step=False):
        data = []
        labels = []
    
        start_index = start_index + history_size
        if end_index is None:
          end_index = len(dataset) - target_size
    
        for i in range(start_index, end_index):
          indices = range(i-history_size, i, step)
          data.append(dataset[indices])
    
          if single_step:
            labels.append(target[i+target_size])
          else:
            labels.append(target[i:i+target_size])
    
        return np.array(data), np.array(labels)
    

This is what we get by applying that function (in your case, you do not need to apply the sampling interval of 6 timesteps, you can set it to 1; this is in case you want to downsample a maybe too long history):

enter image description here

  • function implementation to build the train-eval sets:

      past_history = 720 # historic values to consider for each prediction
      future_target = 72 # number of future values to predict 
      STEP = 6 # sampling frequency (it makes taking 720/6 historic points for each sample)
    
      x_train_multi, y_train_multi = multivariate_data(dataset_standardized, dataset_standardized[:, 1], 0,TRAIN_SPLIT, past_history, future_target, STEP) 
      x_val_multi, y_val_multi = multivariate_data(dataset_standardized, dataset_standardized[:, 1], TRAIN_SPLIT, None, past_history, future_target, STEP)
    

This way you have:

  • Single window of past history : (120, 3)
  • Target temperature to predict : (72,)

For the training phase, we define the batch size and other values and build our training and validation sets:

BATCH_SIZE = 256
BUFFER_SIZE = 10000

train_data_multi = tf.data.Dataset.from_tensor_slices((x_train_multi, y_train_multi))
train_data_multi = train_data_multi.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()

val_data_multi = tf.data.Dataset.from_tensor_slices((x_val_multi, y_val_multi))
val_data_multi = val_data_multi.batch(BATCH_SIZE).repeat()

and defining a simple architecture to begin with, we have:

multi_step_model = tf.keras.models.Sequential()
multi_step_model.add(tf.keras.layers.LSTM(32,
                                          return_sequences=True,  # "Boolean. Whether to return the last output. in the output sequence, or the full sequence"
                                          input_shape=x_train_multi.shape[-2:]))
multi_step_model.add(tf.keras.layers.LSTM(16, activation='relu'))
multi_step_model.add(tf.keras.layers.Dense(72))

multi_step_model.compile(optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0), loss='mae')

for which we can check the performance on the validation set as:

EVALUATION_INTERVAL = 200
EPOCHS = 10

multi_step_history = multi_step_model.fit(train_data_multi, epochs=EPOCHS,
                                          steps_per_epoch=EVALUATION_INTERVAL,
                                          validation_data=val_data_multi,
                                          validation_steps=50) 

having the predictions in red below, after de-scaling the values to its original scale:

for x, y in val_data_multi.take(3):
    x_denormalized = (x[0])*temps_std+temps_mean
    y_denormalized = (y[0])*temps_std+temps_mean
    
    multi_step_plot(x_denormalized, y_denormalized, (multi_step_model.predict(x)[0])*temps_std+temps_mean)

enter image description here

and maybe the most interesting part for your use case, you can try implementing hyperparametrization of some params as hidden layers units as follows with the keras bayesian tuner:

from kerastuner import BayesianOptimization

def build_model_2(hp):
    model = keras.Sequential()
    model.add(keras.layers.LSTM(units=hp.Int('units',min_value=16,
                                        max_value=32,
                                        step=16), 
                                        #activation='relu', 
                                        return_sequences=True, 
                                        input_shape=x_train_multi.shape[-2:]))
    model.add(keras.layers.LSTM(units=hp.Int('units',min_value=16,
                                        max_value=32,
                                        step=16),  
                                        activation='relu'))
    model.add(keras.layers.Dense(72))

    model.compile(loss='mae', optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0),
            
                   metrics=['mae'])
    return model

# define model
bayesian_opt_tuner_2 = BayesianOptimization(
    build_model_2,
    objective='mae',
    max_trials=3,
    executions_per_trial=1,
    directory=os.path.normpath('C:/keras_tuning'),
    project_name='timeseries_temp_ts_test_from_TF_ex_multivar_multistep_2',
    overwrite=True)

EVALUATION_INTERVAL = 200
EPOCHS = 10
bayesian_opt_tuner_2.search(train_data_multi, 
             epochs=EPOCHS,
             validation_data=val_data_multi,
             validation_steps=50,  
             steps_per_epoch=EVALUATION_INTERVAL)

# and selecting the best model: 

    best_MULTIVAR_MULTISTEP_LSTM_model_2 = bayesian_opt_tuner_2.get_best_models(num_models=1)[0]

we have our new model predictions:

enter image description here

which gave an improved MAE on the validation set.

$\endgroup$
4
  • $\begingroup$ Thank you for the detailed answer! Unfortunately I don't understand the data preparation part. How do you organize it? What I do is to take all the samples organized in n_timesteps and feed them into the network so that it predicts n_output which is compared with the n_output points that come after each of the n_timesteps. $\endgroup$
    – Fra
    Jan 30, 2021 at 12:47
  • $\begingroup$ it's true that the data preparation part is not evident at a first glance, you can find the 'multivariate_data ' function used for that together with a new pic in my edited anser :) $\endgroup$
    – German C M
    Jan 30, 2021 at 18:36
  • $\begingroup$ hi @Fra, just to point out that my worked out example is based on tensorflow.org/tutorials/structured_data/time_series covering all types of possible times series cases: single step, multi step, both with multivariate input data; by applying this methodology, you might get some better results ;) $\endgroup$
    – German C M
    Feb 2, 2021 at 17:30
  • $\begingroup$ does anyone have any example of an multistep univariate lstm using timeseriesgenerator? $\endgroup$
    – bardulia
    Jan 30 at 22:46
0
$\begingroup$

Had a similar issue in a project where the model would become strictly dependent on immediately previous temporal data. I imagine this issue is prevalent when dealing with time series, and problems such as predicting something will happen in $T$ time ahead based on data from $T_1$ time behind. The issue with this is that the model can always predict the same as the closest point in time and receive a "good" score. Check this article's header "Time delayed predictions and autocorrelations". The author wrote a follow-up that I also recommend looking at.

My memory notwithstanding, I believe this bias was considerably improved in my project by reshaping the temporal area we trained on to not use a measured value at that point in time, but rather deltas from relevant data's temporal spaces. E.g. if attempting to predict 1 day ahead and you know the pattern is somewhere in the past 1-10 days, create a training set tailored to that. (deltas or sliding deltas from data in that temporal area).

$\endgroup$
4
  • $\begingroup$ Thank you for the answer, unfortunately the article is behind a paywall. On the second part of your answer, could you be more spefcific please? I.e. what would a delta be? $\endgroup$
    – Fra
    Jan 28, 2021 at 15:59
  • $\begingroup$ Try opening it in an incognito window. A delta is the change between values. A delta in your case I could not say as I do not know what your data represents. If you have weather data, and you wish to predict one day ahead, the most meaningful delta I can think of is the change in weather from a few days prior. Thus, my deltas would be the change in weather conditions from 1 day to the most recent, another from 2 days ago till the most recent, etc. There are many ways to construct sliding windows like that, where the goal is to theorise about the most relevant data the algorithm may learn from. $\endgroup$
    – Erlend
    Jan 29, 2021 at 11:53
  • $\begingroup$ @Fra, did you do try opening the article in incognito and understood my example in the above comment? In my opinion this answers your question. $\endgroup$
    – Erlend
    Feb 2, 2021 at 13:05
  • $\begingroup$ I have read the article now! However my problem is not that the prediction follows the true values with a delay sincethe network thinks that the best prediction for t+1 is t. When I predict a step forward my model predicts without a delay of a step like it is described in the article for random walks. The problem is that when I try to predict more than one point the model simply doesn't reproduce the data it doesn't even follow, its just a mess $\endgroup$
    – Fra
    Feb 2, 2021 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.