-1
$\begingroup$

I would like to pass a series of values $x_1, x_2...$ as input to the model to predict $y_1 = sin(x_1), y_2 = sin(x_2)...$

-I created dataset: $x=[0.1,0.2,...]$ and $y=[sin(0.1),sin(0.2),...]$

-I normalize x in $[0,1]$ (not y because it has range $[-1,+1]$.

-I split x and y in: x_train/y_train, x_val/y_val, x_test/y_test

-I pass x_train and y_train to fit model lstm

it doesn't even work for the training set. (Maybe for the test set it cannot work because x_test is out of range of x_train?)

I set time_steps = 70 because I need to set more than a sin period, I think

I tried to fit for more than 20 epochs but the train/validation loss not changed..

import sys
import os
import numpy as np
import math
import pandas as pd
from matplotlib import pyplot as plt

from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

from keras.models import Sequential, load_model
from keras.layers import LSTM, Dense, Flatten, Dropout
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras import backend as K
K.clear_session() 
# hyper-parametri
params = {
    "batch_size": 20,
    "epochs": 20,
    "time_steps": 70,
}

OUTPUT_PATH = "..."
TIME_STEPS = params["time_steps"]
BATCH_SIZE = params["batch_size"]

n_periods = 20
n_points = int(2*math.pi*n_periods)*10+1
x = np.linspace(0, int(2*math.pi*n_periods), n_points)
y = np.zeros(n_points)

for i in range(0, n_points):
    y[i] += math.sin(x[i])

Whole Dataset

x = (x - np.min(x))/np.ptp(x) #normalize

def create_timeseries(arr1, arr2):
    # build univariate time series
    dim_0 = len(arr1) - TIME_STEPS

    x = np.zeros((dim_0, TIME_STEPS))
    y = np.zeros((dim_0,))

    for i in range(dim_0):
        x[i] = arr1[i:TIME_STEPS+i] #TIME_STEPS+i non compreso
        y[i] = arr2[TIME_STEPS+i-1]
        #print(x[i], y[i])
    print("length of time-series i/o",x.shape,y.shape)
    return x, y

def adjust_dataset(mat, batch_size):
    # eliminates the excess dataset portion

    no_samples_to_drop = mat.shape[0] % batch_size

    if(no_samples_to_drop > 0):
        return mat[:-no_samples_to_drop]
    else:
        return mat

x_ts, y_ts = create_timeseries(x, y)
# reshape da [samples, timesteps] in [samples, timesteps, features]
n_features = 1
x_ts = x_ts.reshape((x_ts.shape[0], x_ts.shape[1], n_features))

len_train = int(len(x_ts)*80/100)
len_val = int(len(x_ts)*10/100)
#DATASET DI TRAINING 80%
x_train = x_ts[0:len_train,:,:]
y_train = y_ts[0:len_train]
#DATASET DI VALIDATION 10%
x_val = x_ts[len_train:len_train+len_val,:,:]
y_val = y_ts[len_train:len_train+len_val]
#DATASET DI TEST 10%
x_test = x_ts[len_train+len_val:,:,:]
y_test = y_ts[len_train+len_val:]

x_train = adjust_dataset(x_train, BATCH_SIZE)
y_train = adjust_dataset(y_train, BATCH_SIZE)
x_val = adjust_dataset(x_val, BATCH_SIZE)
y_val = adjust_dataset(y_val, BATCH_SIZE)
x_test = adjust_dataset(x_test, BATCH_SIZE)
y_test = adjust_dataset(y_test, BATCH_SIZE)

print(x_train.shape, y_train.shape)
print(x_val.shape, y_val.shape)
print(x_test.shape, y_test.shape)

def create_model():
    model = Sequential()
    model.add(LSTM(30, input_shape=(TIME_STEPS, x_train.shape[2]), return_sequences=True))
    model.add(LSTM(30, input_shape=(TIME_STEPS, x_train.shape[2])))
    model.add(Dense(1))

    model.compile(optimizer='adam', loss='mse')
    return model 

model = create_model()
model.summary()

history = model.fit(x_train, y_train, epochs=params["epochs"], verbose=2, batch_size=BATCH_SIZE, shuffle=False,validation_data=(x_val, y_val))

Results:

Test set MSE: 0.49853

MSE loss training/validation set Training set prediction

$\endgroup$

2 Answers 2

0
$\begingroup$

I have two observations about your code:

  • Your output is in the [-1, 1] range, therefore you should put tanh activation at the last layer. If you don't specify an activation you'll have a linear one; with tanh you force your output to be in the right range.

  • You don't need two LSTM() layers with 30 cells each. You have specified way too many parameters for a simple, easy to forecast function such as sin(x). If you have too many cells/parameters for a relatively simple information, the result is that most of your cells won't have enought to learn from. I suggest you to keep one layer, with a much lower number of cells. It's also going to be much faster to train, so you can experiment more: try with very few cells, and see if increasing their number improves the results or not.

$\endgroup$
3
  • $\begingroup$ I followed your advice but it cannot predict anyway... I tried different models but the results are the same... I tried to set timestep = 1 and my PREDICTION ON TRAINING SET plot start from 0 $\endgroup$
    – Izanagi
    Oct 4, 2019 at 16:59
  • $\begingroup$ I just noticed that you set input_shape in both LSTM layers... you should set it only for the input layer. Did you try to change it? $\endgroup$
    – Leevo
    Oct 5, 2019 at 14:50
  • $\begingroup$ you're right i made a mistake, thanks $\endgroup$
    – Izanagi
    Oct 6, 2019 at 15:19
0
$\begingroup$

I guess that too many periods make all sequences too many near. If you reduce the period to 1 the accuracy improves dramatically:

import sys
import os
import numpy as np
import math
import pandas as pd
from matplotlib import pyplot as plt

from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

from keras.models import Sequential, load_model
from keras.layers import LSTM, Dense, Flatten, Dropout
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras import backend as K
K.clear_session() 
# hyper-parametri
params = {
    "batch_size": 20,
    "epochs": 20,
    "time_steps": 30,
}

OUTPUT_PATH = "..."
TIME_STEPS = params["time_steps"]
BATCH_SIZE = params["batch_size"]

#1 periodo
n_periods = 1
n_points = int(2*math.pi*n_periods)*100+1
x = np.linspace(0, int(2*math.pi*n_periods), n_points)
y = np.zeros(n_points)

for i in range(0, n_points):
    y[i] += math.sin(x[i])

def create_timeseries(arr1, arr2):
    # build univariate time series
    dim_0 = len(arr1) - TIME_STEPS
    print("dim0: ",dim_0)

    x = np.zeros((dim_0, TIME_STEPS))
    y = np.zeros((dim_0,))

    for i in range(dim_0):
        x[i] = arr1[i:TIME_STEPS+i] #TIME_STEPS+i non compreso
        y[i] = arr2[TIME_STEPS+i-1]
        print(x[i], y[i])
    print("length of time-series i/o",x.shape,y.shape)
    return x, y

def adjust_dataset(mat, batch_size):
    # eliminates the excess dataset portion

    no_samples_to_drop = mat.shape[0] % batch_size

    if(no_samples_to_drop > 0):
        return mat[:-no_samples_to_drop]
    else:
        return mat

x_ts, y_ts = create_timeseries(x, y)

# reshape da [samples, timesteps] in [samples, timesteps, features]
n_features = 1
x_ts = x_ts.reshape((x_ts.shape[0], x_ts.shape[1], n_features))

len_train = int(len(x_ts)*80/100)
len_val = int(len(x_ts)*10/100)
#DATASET DI TRAINING 80%
x_train = x_ts[0:len_train,:,:]
y_train = y_ts[0:len_train]
#DATASET DI VALIDATION 10%
x_val = x_ts[len_train:len_train+len_val,:,:]
y_val = y_ts[len_train:len_train+len_val]
#DATASET DI TEST 10%
x_test = x_ts[len_train+len_val:,:,:]
y_test = y_ts[len_train+len_val:]

print(x_train.shape, y_train.shape)
print(x_val.shape, y_val.shape)
print(x_test.shape, y_test.shape)

def create_model():
    model = Sequential()
    model.add(LSTM(67, input_shape=(TIME_STEPS, x_train.shape[2])))
    model.add(Dense(1))

    model.compile(optimizer='adam', loss='mse')
    return model 

model = create_model()
model.summary()

history = model.fit(x_train, y_train, 
                    epochs=params["epochs"], 
                    verbose=2, 
                    batch_size=BATCH_SIZE, 
                    shuffle=False,
                    validation_data=(x_val, y_val))

The result:

Layer (type)                 Output Shape              Param #   
=================================================================
lstm_1 (LSTM)                (None, 67)                18492     
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 68        
=================================================================
Total params: 18,560
Trainable params: 18,560
Non-trainable params: 0
_________________________________________________________________
WARNING:tensorflow:From /anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Train on 456 samples, validate on 57 samples
Epoch 1/20
 - 2s - loss: 1.1326 - val_loss: 1.3267
Epoch 2/20
 - 1s - loss: 0.5891 - val_loss: 0.3672
Epoch 3/20
 - 1s - loss: 0.4792 - val_loss: 0.1675
Epoch 4/20
 - 1s - loss: 0.3520 - val_loss: 0.0126
Epoch 5/20
 - 1s - loss: 0.2238 - val_loss: 0.1767
Epoch 6/20
 - 1s - loss: 0.0611 - val_loss: 0.1351
Epoch 7/20
 - 1s - loss: 0.0752 - val_loss: 0.0624
Epoch 8/20
 - 1s - loss: 0.0384 - val_loss: 0.0170
Epoch 9/20
 - 1s - loss: 0.0554 - val_loss: 0.0958
Epoch 10/20
 - 1s - loss: 0.1817 - val_loss: 0.0846
Epoch 11/20
 - 1s - loss: 0.0192 - val_loss: 0.1071
Epoch 12/20
 - 1s - loss: 0.1207 - val_loss: 0.2081
Epoch 13/20
 - 1s - loss: 0.1679 - val_loss: 0.0095
Epoch 14/20
 - 1s - loss: 0.0276 - val_loss: 0.0680
Epoch 15/20
 - 1s - loss: 0.0347 - val_loss: 0.0960
Epoch 16/20
 - 1s - loss: 0.0551 - val_loss: 0.0148
Epoch 17/20
 - 1s - loss: 0.0234 - val_loss: 0.0814
Epoch 18/20
 - 1s - loss: 0.0129 - val_loss: 0.0258
Epoch 19/20
 - 1s - loss: 0.0187 - val_loss: 0.0210
Epoch 20/20
 - 1s - loss: 0.0521 - val_loss: 0.1205

This way one LSTM layer is enough to get a good behaviour.

$\endgroup$
3
  • $\begingroup$ that is because i go back 70 time steps, so the first sine predict is the 70-th sine $\endgroup$
    – Izanagi
    Oct 4, 2019 at 16:58
  • $\begingroup$ Try this new config. When you have 20 periods these 70 time series are too many similar (clearly identical but a shift phase) that LSTM is not good to differentiate. With one period all signals are clearly different. $\endgroup$ Oct 5, 2019 at 16:17
  • $\begingroup$ Thanks a lot for your solution. I think the problem is that although the problem may seem trivial, a neural network cannot learn to output a sine wave for any input (in fact the test set does not work in any case ...), but can only train itself to predict the training set in this case, and, by giving it a single period of input, the network has enough weights to learn all the training set. $\endgroup$
    – Izanagi
    Oct 6, 2019 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.