1
$\begingroup$

I am doing some comparison of RNN with other methodologies to check if the RNNs can improve some more "classical" models. In fact, I am doing it with multiple features (similar to what was done here) with a time-series function where the target is a function of other variables. The objective is to predict the target evolution (forecast for different time-horizons) having as input the target and also features that might be important for that evolution (are function of the target). I made the following time-series dataset as an example.

import numpy as np
import pandas as pd

# Create dataset
time = np.arange(1000)
f_1 = np.sin(time)
f_2 = np.cos(time)
f_3 = np.tan(time)
target = np.roll(f_1, 1) + np.roll(f_2,2) + np.roll(f_3,3)
df = pd.DataFrame({'target': target, 'f_1': f_1, 'f_2': f_2, 'f_3': f_3})

So, I create a dataset where I make samples with 4 variables (target + features) using the previous 60 values (n_prev) to predict the following 20 values (n_next).

def _load_data_multiVar(df, n_prev=60, n_next=20):  
    docX, docY = [], []
    for i in range(len(df)-n_prev-n_next):
        docX.append(df.iloc[i:i+n_prev].as_matrix())
        docY.append(df.iloc[i+n_prev:i+n_prev+n_next]['target'].as_matrix())
    alsX = np.array(docX)
    alsY = np.array(docY)
    return alsX, alsY

This creates the dimensions (20% test size) of: X_train: (720, 60, 4) - y_train: (720, 20) X_test: (120, 60, 4) - y_test: (120, 20)

So now, I can predict the following 20 values with RNN altogether, or traing NN for each time horizon and merge later on. So that's what I've tried (trying to make a similar complex models) and the results shown that the NN are much better than the RNN, also for the time-horizon 0.

RNN multivariable forecast for different time horizons NN multivariable forecast for different time horizons

The number of the legend is the time-horizon forecast.

As it can be seen, the RNN Root Mean Squared Error is surprisingly high, which is what I am not correctly understanding as RNN are using LSTMs where input is a repetitive function with features that should improve its knowledge. I might be doing something wrong, or maybe the RNNs are not good for this kind of problems.

Why is the RNN RMSE higher than the NN RMSE, in this case?

Here is the full working code to check the used RNN and NN parameter/configuration:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from keras.models import Sequential
from keras.layers.core import Dense, Activation, Dropout
from keras.layers.recurrent import LSTM

###############################################################################
def _load_data_multiVar(df, n_prev=60, n_next=20):  
    docX, docY = [], []
    for i in range(len(df)-n_prev-n_next):
        docX.append(df.iloc[i:i+n_prev].as_matrix())
        docY.append(df.iloc[i+n_prev:i+n_prev+n_next]['target'].as_matrix())
    alsX = np.array(docX)
    alsY = np.array(docY)
    return alsX, alsY

###############################################################################

def Model_NN_keras(input_features):
    from keras.models import Sequential
    from keras.layers.core import Dense, Activation, Dropout
    from keras.optimizers import Adam

    hidden_1_size = 128
    hidden_2_size = 128

    model = Sequential()
    model.add(Dense(hidden_1_size, input_shape=(input_features,)))
    model.add(Dropout(0.2)) 
    model.add(Dense(hidden_2_size, activation='relu'))
    model.add(Dense(1, activation='linear'))
    model.compile(loss='mean_squared_error',    optimizer="rmsprop")
    print model.summary()
    return model

def Model_RNN_keras(input_features, window_size, output_features):
    hidden_neurons = 300
    model = Sequential()
    model.add(LSTM(hidden_neurons, return_sequences=False, input_shape=(window_size,input_features)))
    model.add(Dense(output_features))
    model.add(Activation("linear"))
    model.compile(loss="mean_squared_error", optimizer="rmsprop")
    print model.summary()
    return model


###############################################################################
if __name__ == "__main__":

    # Create dataset
    time = np.arange(1000)
    f_1 = np.sin(time)
    f_2 = np.cos(time)
    f_3 = np.tan(time)
    target = np.roll(f_1, 1) + np.roll(f_2,2) + np.roll(f_3,3)
    df = pd.DataFrame({'target': target, 'f_1': f_1, 'f_2': f_2, 'f_3': f_3})

    # Split train/test [X(num. samples, prev. window size for trend, num. features), y(num. samples, next window size to predict)]
    test_size = 0.2
    ntrn = int(len(df) * (1 - test_size))
    X_train, y_train = _load_data_multiVar(df.iloc[0:ntrn], n_prev=60, n_next=20)
    X_test, y_test = _load_data_multiVar(df.iloc[ntrn:], n_prev=60, n_next=20)
    print 'X_train: ', X_train.shape, ' - y_train: ', y_train.shape
    print 'X_test: ', X_test.shape, ' - y_test: ', y_test.shape

    ########## NN ##########
    # Plot
    f, axarr = plt.subplots(7, sharex=True, figsize=(16,7), dpi=100)
    index = 0
    for tmhrzn in range(20):

        if(tmhrzn % 3 != 0):
            continue

        X_train_NN = X_train.reshape(X_train.shape[0],-1)
        X_test_NN = X_test.reshape(X_test.shape[0],-1)
        y_train_NN = y_train[:,tmhrzn]
        y_test_NN = y_test[:,tmhrzn]
        print 'X_train_NN: ', X_train_NN.shape, ' - y_train_NN: ', y_train_NN.shape
        model = Model_NN_keras(X_train.shape[1]*X_train.shape[2])
        model.fit(X_train_NN, y_train_NN, batch_size=64, epochs=10, validation_split=0.05)
        y_pred_NN = np.squeeze(model.predict(X_test_NN))    
        print 'y_pred_NN: ', y_pred_NN.shape, ' - y_test_NN: ', y_test_NN.shape


        axarr[index].plot(range(len(y_pred_NN)), y_pred_NN, color='g', ls='--', lw=1, label='Pred_%.2d' % tmhrzn)
        axarr[index].plot(range(len(y_test_NN)), y_test_NN, color='k', ls='-', lw=1, label='Real_%.2d' % tmhrzn)
        axarr[index].legend()
        axarr[index].set_xlim([-5,135])
        axarr[index].set_ylim([-20,30])
        axarr[index].text(0., 10., 'RMSE: %.3f' % np.sqrt(np.sum((y_pred_NN - y_test_NN) ** 2)/len(y_test_NN)) )
        if(index != 6):
            axarr[index].set_xticklabels([])
        index += 1

    plt.subplots_adjust(hspace=None)
    plt.suptitle('NN')
    plt.show()
    exit()

    ########## RNN ##########
    model = Model_RNN_keras(X_train.shape[2], X_train.shape[1], y_train.shape[1])
    model.fit(X_train, y_train, batch_size=64, epochs=10, validation_split=0.05)
    y_pred = model.predict(X_test)
    print 'y_pred: ', y_pred.shape, ' - y_test: ', y_test.shape
    rmse = np.sqrt(((y_pred - y_test) ** 2).mean(axis=0))
    print 'rmse: ', rmse, rmse.shape

    # Plot
    f, axarr = plt.subplots(7, sharex=True, figsize=(16,7), dpi=100)
    index = 0
    for tmhrzn in range(20):

        if(tmhrzn % 3 != 0):
            continue

        axarr[index].plot(range(y_pred.shape[0]), y_pred[:,tmhrzn], color='g', ls='--', lw=1, label='Pred_%.2d' % tmhrzn)
        axarr[index].plot(range(y_test.shape[0]), y_test[:,tmhrzn], color='k', ls='-', lw=1, label='Real_%.2d' % tmhrzn)
        axarr[index].legend()
        axarr[index].set_xlim([-5,135])
        axarr[index].set_ylim([-20,30])
        axarr[index].text(0., 10., 'RMSE: %.3f' % rmse[tmhrzn] )
        if(index != 6):
            axarr[index].set_xticklabels([])
        index += 1

    plt.subplots_adjust(hspace=None)
    plt.suptitle('RNN')
    plt.show()
$\endgroup$
2
  • $\begingroup$ What's the specific question? "Why is RMSE so high?", "How can I reduce the RMSE?" ...? Please ask one specific question. $\endgroup$ – Mephy Aug 4 '18 at 13:02
  • $\begingroup$ The question is why RNNs RMSE is so high when are methodologies supposed to work better than NNs for time evolution problems $\endgroup$ – iblasi Aug 5 '18 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.