0
$\begingroup$

I've been working on a multivariate LSTM model for time series forecasting, but I'm encountering an issue where the predicted output doesn't exhibit enough variability. The predictions tend to be too smooth or flat, particularly after the first predicted point. Here's anoverview of my model architecture in Tensorflow 2.10 (implementing the cuDNN implementation of LSTM):

model.add(tf.keras.layers.InputLayer(input_shape=(60,25)))
model.add(LSTM(256,return_sequences=True))
model.add(tf.keras.layers.LayerNormalization())
model.add(Dropout(0.2))
model.add(LSTM(256,return_sequences=True))
model.add(tf.keras.layers.LayerNormalization())
model.add(Dropout(0.2))
model.add(LSTM(256,return_sequences=False))
model.add(tf.keras.layers.LayerNormalization())
model.add(Dropout(0.2))
model.add(Dense(30 * 3,activation=tf.keras.layers.LeakyReLU(alpha=0.1))
model.add(Reshape([30, 3]))

What I am trying to achieve is to have the output layer predict 90 points, which will be further reshaped to three variables. Regarding the data:

  • 670 000 rows, 25 features
  • using 60 past points - prediction of 30 points into the future for 3 target features
  • dataset is first split into training, validation and test (70:20:10) using sliding window
  • each sliding window is one row shifted - shift 1
  • using StandardScaler for scaling as I have couple anomalies in the data I would like to detect afterwards on the results
  • The graph of predicted vs. true values can be seen for one feature below:

enter image description here

My question is: Did I correctly handle the last layers from the LSTM? My goal is for having one-shot prediction of 30 values for 3 features.

What I tried:

  • Hyperparameter tuning containing
    • Number of layers
    • Optimizer (RMSprod, Adam, SGD)
    • Layers - LSTM/GRU
    • Number of units in LSTM/GRU
    • Dropout Rate Chance
    • Learning rate (0.01, 0.001, 0.005)
    • Batch size of sliding windows (64, 128, 256)
  • Warmup for first 4 layers
  • Using other scaling methods (Quantile, MinMaxScaler, PowerTransformer)

Link to documentation of LSTM https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.