Multivariate LSTM RMSE value is getting very high

Question

I want to predict a time series with multiple variables. I am using Keras's LSTM class.

Here is my data set description :

I want to predict var1(t-1) and my X variables are var3(t-1) , var4(t-1) , var5(t-1) , var6(t-1) and var7(t-1).

And this is how my model configuration looks like :

{
    "data": {

        "columns": [            
            "var1(t-1)",
            "var3(t-1)",
            "var4(t-1)",          
            "var5(t-1)",
            "var6(t-1)",
            "var7(t-1)" 
        ],
        "sequence_length": 5,
        "train_test_split": 0.80,
        "normalise": false
    },
    "training": {
        "epochs": 100,
        "batch_size":10 
    },
    "model": {
        "loss": "mse",
        "optimizer": "nadam",
        "save_dir": "saved_models_multi",
        "layers": [
            {
                "type": "lstm",
                "neurons": 200,
                "input_timesteps": 5,
                "input_dim": 5,
                "return_seq": true
            },
            {
              "type":"batch_norm"  

            },  
            {
                "type": "dropout",
                "rate": 0.4
            },

            {
                "type": "lstm",
                "neurons": 200,
                "return_seq": true
            },
            {
              "type":"batch_norm"  

            },

            {
                "type": "dropout",
                "rate": 0.4
            },
            {
                "type": "dense",
                "neurons": 50,
                "activation": "sigmoid"
            },

             {
              "type":"batch_norm"  

            },

            {
                "type": "dropout",
                "rate": 0.2
            },

            {
                "type": "lstm",
                "neurons": 200,
                "return_seq": false
            },
            {
              "type":"batch_norm"  

            },

            {
                "type": "dropout",
                "rate": 0.4
            },
            {
                "type": "dense",
                "neurons": 25,
                "activation": "sigmoid"
            },

             {
              "type":"batch_norm"  

            },

            {
                "type": "dropout",
                "rate": 0.2
            },

            {
                "type": "dense",
                "neurons": 1,
                "activation": "linear"
            }
        ]
    }
}

This is number of parameters in each layer and output shape :

Layer (type)                 Output Shape              Param #   
=================================================================
lstm_1 (LSTM)                (None, 5, 200)            164800    
_________________________________________________________________
batch_normalization_1 (Batch (None, 5, 200)            800       
_________________________________________________________________
dropout_1 (Dropout)          (None, 5, 200)            0         
_________________________________________________________________
lstm_2 (LSTM)                (None, 5, 200)            320800    
_________________________________________________________________
batch_normalization_2 (Batch (None, 5, 200)            800       
_________________________________________________________________
dropout_2 (Dropout)          (None, 5, 200)            0         
_________________________________________________________________
dense_1 (Dense)              (None, 5, 50)             10050     
_________________________________________________________________
batch_normalization_3 (Batch (None, 5, 50)             200       
_________________________________________________________________
dropout_3 (Dropout)          (None, 5, 50)             0         
_________________________________________________________________
lstm_3 (LSTM)                (None, 200)               200800    
_________________________________________________________________
batch_normalization_4 (Batch (None, 200)               800       
_________________________________________________________________
dropout_4 (Dropout)          (None, 200)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 25)                5025      
_________________________________________________________________
batch_normalization_5 (Batch (None, 25)                100       
_________________________________________________________________
dropout_5 (Dropout)          (None, 25)                0         
_________________________________________________________________
dense_3 (Dense)              (None, 1)                 26        
=================================================================
Total params: 704,201
Trainable params: 702,851
Non-trainable params: 1,350
_________________________________________________________________
Time taken: 0:00:01.914407

But , I am seeing RMSE value for test data is getting very high. This is my train and validation loss graph.

What am I doing wrong here ? Is my model suffering from overfitting problem ?

Please help .

Answer 1

You definitely have an overfitting problem. I have few observations to look about your model:

1. Reduce the number of LSTM layers drastically. RNNs are different from CNNs, and usually don't benefit from stacking several layers on top of each other (their "depth" is more in the sequentiality of the signal). Usually you never need more than one, two at most. I suggest you to take a look at this very good tutorial on time series forecasting on the official TensorFlow website.

2. Do not use batchnorm after LSTM layers. They are sequential in nature, and I'd leave the output signal as intact as possible. Batchnorm subtracts the mean and scales (usually on the last axis of the tensor) and in that way you'd loose lots of information coming from the sequence of outputs.

3. Do not use simple dropout. LSTM layers have recurrent dropout that was designed specifically for recurrent layers. IMHO, do no use dropout at all with RNNs. Once again, sequential information would be distorted/lost. If you want to add regularization, add dropout to the last Dense layers of the Network instead.

In order to fight overfitting you can start simpler:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import GRU, Dense

RNN = Sequential([
    GRU(n_lstm_units, input_shape),
    Dense(1)
])

Something like that can do the job better than a heavy network. (NB: I used GRU because they have less parameters (they are faster to train) and sometimes perform equally good.)

Then you can try to progressively to build up fancier architectures. For example, add more Dense layers and regularization between them, change parameter initialization, batch size, learning rate, ... you name it.

Also, please consider using vanilla Dense networks instead of RNNs. If the number of input time steps is small (5 if I understood correctly) then you probably won't need the complexity of Recurrent architectures, and Dense layers can do the job even better (since they are more fully connected with subsequent layers).

Answer 2

A high RMSE on the test set with a small RMSE on the train set is a sign of overfitting. Your plot looks weird, as there's no sign of overfitting on the validation set (I suppose that the label test means validation following your text). This might be caused by:

• your validation data doesn't represent your test data, e.g. they come from different distributions, the train/validation/test splits have not been carried out correctly, etc.
• you are overfitting the validation set, that is you have tweaked the parameters so many times based on what you have seen on the validation set that the RMSE on that specific set is almost perfect, but it doesn't generalize well on other sets

I suggest to use the usual techniques to prevent overfitting: start with a simpler architecture (fewer layers and nodes) and build from there, use cross-validation to adjust the dropout probability, perhaps include other forms of regularization, and so on.