# Loss function returns nan on time series dataset using tensorflow

This was the follow up question of Prediction on timeseries data using tensorflow.

I have an input and output of below format.

(X) = [[ 0  1  2]
[ 1  2  3]]

y   = [ 3  4 ]


Its a timeseries data. The task is to predict the next number. Basically input was crafted by the below snippet

 def split_sequence(arr,timesteps):
arr_len = len(arr)
X,y = [],[]
for i in range(arr_len):
end_idx = i + timesteps
if end_idx > arr_len-1:
break
input_component,output_component =  arr[i:end_idx],arr[end_idx]
X.append(input_component)
y.append(output_component)

return np.array(X), np.array(y)


Now i want to train the model on the input and predict the next number. For instance x = [81,82,83] and the predicted output would be y = 84. In the previous problem, i had confronted the shape issue. Fortunately, i got a quick fill.

Now, when i am training the model,I observe my mse values are nan.

My implementation in keras is working but not in tensorflow. Most of the solutions in stackoverflow was pointing out to learning rate. Irrespective of giving different learning rate, my mse values are still nan. I have gone through many times reading my implementation but nothing turned up. Below is my tensorflow implementation.

 import tensorflow as tf
import numpy as np

# Create a dataset
def split_sequence(arr,timesteps):
arr_len = len(arr)
X,y = [],[]
for i in range(arr_len):
end_idx = i + timesteps
if end_idx > arr_len-1:
break
input_component,output_component = arr[i:end_idx],arr[end_idx]
X.append(input_component)
y.append(output_component)

return np.array(X), np.array(y)

# data generator
def generate_batch(X,y,batch_size):
m = X.shape
indexes = range(m)
n_batches = m // batch_size
for batch_index in np.array_split(indexes,n_batches):
yield X[batch_index],y[batch_index]

# parameters
n_inputs = 3
n_epochs = 400
batch_size = 500
learning_rate = 0.0005
n_steps = 3
input, output = split_sequence(range(10000),n_steps)

# input and output and theta variable
X = tf.placeholder(tf.float32,shape=(None,n_inputs),name='X')
y = tf.placeholder(tf.float32,shape=(None),name='y')
theta = tf.Variable(tf.random_uniform([n_steps,1],-1.0,1.0),name='theta')

# Predictions and the residual
y_predictions = tf.matmul(X,theta,name='predictions')
error = y_predictions - y

mse = tf.reduce_mean(tf.square(error),name='mse')

# Training the algorithm
training_op = optimizer.minimize(mse)

# initialize the variables
init = tf.global_variables_initializer()

with tf.Session() as session:
# create  a session

session.run(init)
for epoch in range(n_epochs):
for X_batch,y_batch in generate_batch(input,output,batch_size):
#print(X_batch,y_batch)
mse_val,_ =  session.run([mse,training_op],feed_dict={X:X_batch,y:y_batch})

if epoch % 10 == 0:
print('epoch',epoch,'MSE=',mse_val)


And the output i got was

epoch 0 MSE= nan

epoch 10 MSE= nan

epoch 20 MSE= nan

Any help is greatly appreciated. Thanks

This is because the the input values are not normalised to a standard scale. X consists of values ranging from 0 to 9999. This would make the training unstable. If the idea is to train the net to predict the sequence that you described, an easy fix is to feed it smaller values in the training data. For instance, feed it the input,

output = split_sequence(range(10),n_steps)


Along with a batch size of say, 5. It will learn to generalize to higher numbers as well.

With the changes made and the following for loop, the output is as expected.

for epoch in range(n_epochs):
for X_batch, y_batch in generate_batch(input, output, batch_size):
# print(X_batch,y_batch)
mse_val, _ = session.run([mse, training_op], feed_dict={
X: X_batch, y: y_batch})

if epoch % 100 == 0:
print('epoch', epoch, 'MSE=', mse_val)
error_list.append(mse_val)

X_test = np.array([[45, 46, 47], [57, 58, 59], [1000, 1001, 1002]])
y_test = session.run(y_predictions, feed_dict={X:X_test})

print(y_test) # Should be 48, 60, 1003

plt.plot(error_list)
plt.show()


Final y_test is

[[  47.997036]
[  59.996166]
[1002.9279  ]]


If not this change, another change that you can make is to use the Adam Optimiser instead of SGD. It will make the learning more stable and it works even with bigger values.

• changing the optimizer to adam worked for me . Oct 20 '18 at 15:22