0
$\begingroup$

This was the follow up question of Prediction on timeseries data using tensorflow.

I have an input and output of below format.

(X) = [[ 0  1  2]
       [ 1  2  3]] 

  y   = [ 3  4 ]

Its a timeseries data. The task is to predict the next number. Basically input was crafted by the below snippet

 def split_sequence(arr,timesteps):
     arr_len = len(arr)
     X,y = [],[]
     for i in range(arr_len):
       end_idx = i + timesteps
       if end_idx > arr_len-1:
          break
       input_component,output_component =  arr[i:end_idx],arr[end_idx]
       X.append(input_component)
       y.append(output_component)

     return np.array(X), np.array(y)

Now i want to train the model on the input and predict the next number. For instance x = [81,82,83] and the predicted output would be y = 84. In the previous problem, i had confronted the shape issue. Fortunately, i got a quick fill.

Now, when i am training the model,I observe my mse values are nan.

My implementation in keras is working but not in tensorflow. Most of the solutions in stackoverflow was pointing out to learning rate. Irrespective of giving different learning rate, my mse values are still nan. I have gone through many times reading my implementation but nothing turned up. Below is my tensorflow implementation.

 import tensorflow as tf
 import numpy as np

   # Create a dataset
  def split_sequence(arr,timesteps):
    arr_len = len(arr)
    X,y = [],[]
    for i in range(arr_len):
       end_idx = i + timesteps
       if end_idx > arr_len-1:
          break
       input_component,output_component = arr[i:end_idx],arr[end_idx]
       X.append(input_component)
       y.append(output_component)

    return np.array(X), np.array(y)

 # data generator
def generate_batch(X,y,batch_size):
    m = X.shape[0]
    indexes = range(m)
    n_batches = m // batch_size
    for batch_index in np.array_split(indexes,n_batches):
       yield X[batch_index],y[batch_index]


  # parameters
   n_inputs = 3
   n_epochs = 400
   batch_size = 500
   learning_rate = 0.0005
   n_steps = 3
   input, output = split_sequence(range(10000),n_steps)


   # input and output and theta variable
   X = tf.placeholder(tf.float32,shape=(None,n_inputs),name='X')
   y = tf.placeholder(tf.float32,shape=(None),name='y')
   theta = tf.Variable(tf.random_uniform([n_steps,1],-1.0,1.0),name='theta')

   # Predictions and the residual
   y_predictions = tf.matmul(X,theta,name='predictions')
   error = y_predictions - y

   mse = tf.reduce_mean(tf.square(error),name='mse')

   # Training the algorithm
   optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
    training_op = optimizer.minimize(mse)

   # initialize the variables
   init = tf.global_variables_initializer()


  with tf.Session() as session:
  # create  a session

     session.run(init)
     for epoch in range(n_epochs):
        for X_batch,y_batch in generate_batch(input,output,batch_size):
          #print(X_batch,y_batch)
          mse_val,_ =  session.run([mse,training_op],feed_dict={X:X_batch,y:y_batch})

          if epoch % 10 == 0:
             print('epoch',epoch,'MSE=',mse_val)

And the output i got was

epoch 0 MSE= nan

epoch 10 MSE= nan

epoch 20 MSE= nan

Any help is greatly appreciated. Thanks

$\endgroup$
1
$\begingroup$

This is because the the input values are not normalised to a standard scale. X consists of values ranging from 0 to 9999. This would make the training unstable. If the idea is to train the net to predict the sequence that you described, an easy fix is to feed it smaller values in the training data. For instance, feed it the input,

output = split_sequence(range(10),n_steps)

Along with a batch size of say, 5. It will learn to generalize to higher numbers as well.

With the changes made and the following for loop, the output is as expected.

for epoch in range(n_epochs):
    for X_batch, y_batch in generate_batch(input, output, batch_size):
        # print(X_batch,y_batch)
        mse_val, _ = session.run([mse, training_op], feed_dict={
                                 X: X_batch, y: y_batch})

    if epoch % 100 == 0:
        print('epoch', epoch, 'MSE=', mse_val)
        error_list.append(mse_val)

X_test = np.array([[45, 46, 47], [57, 58, 59], [1000, 1001, 1002]])
y_test = session.run(y_predictions, feed_dict={X:X_test})

print(y_test) # Should be 48, 60, 1003

plt.plot(error_list)
plt.show()

Final y_test is

[[  47.997036]
 [  59.996166]
 [1002.9279  ]]

If not this change, another change that you can make is to use the Adam Optimiser instead of SGD. It will make the learning more stable and it works even with bigger values.

| improve this answer | |
$\endgroup$
  • $\begingroup$ changing the optimizer to adam worked for me . $\endgroup$ – James K J Oct 20 '18 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.