# Training LSTM Recurrent Network in TensorFlow

I have trained RNN's before "by hand" using basic tools like Numpy or BLAS, but I am having trouble getting a simple RNN to converge in TF. Full Code

I tried standard things like adjusting the learning rate, the momentum, and adding noise to the gradient, but I am concerned that I don't understand what TF is doing underneath.

In a couple tutorials, it appeared that in TF you train the networks like Echo State Networks where the hidden state is evolved on random interconnections to train up a reservoir, then you use a linear (or non-linear) transformation to map the hidden state onto your labels.

I examined the gradients that Tensorflow uses to train, and it appears that it is training the internal state transition matrices as one would expect if using "Back-Propagation Through Time", which I can't tell if it is trying to use.

Can you help me understand what I am doing/understanding incorrectly with regards to training RNN's in TF?

cell_layer = tf.contrib.rnn.BasicLSTMCell(state_size, state_is_tuple=True)
cell = tf.contrib.rnn.MultiRNNCell([cell_layer]*1)
x, y_ = tf.placeholder(tf.float32, shape=(batch_size,1)), tf.placeholder(tf.float32, shape=(batch_size,1))
outputs, states = tf.nn.dynamic_rnn(cell, tf.expand_dims(x, -1), dtype=tf.float32)# init_state)

W = tf.Variable(tf.random_uniform((state_size, 1), -1.0, 1.0))
b = tf.Variable(tf.random_uniform((1, 1), -1.0, 1.0))
y = tf.matmul(tf.reshape(outputs, (-1, state_size)), W) + b

loss = tf.reduce_mean(tf.square(y - y_))
opt = tf.train.MomentumOptimizer(1e-3, 0.9)
grad = [(g + tf.random_uniform(v.shape, -1.0, 1.0) * tf.reduce_mean(tf.abs(v))*noise, v)


EDIT: Loss plot

I am less concerned about the loss than I am about the fact that there appear to not be any dynamics in between iterations. It appears to converge to the average. Below are some of the later losses and a part of the sequence to be predicted.

Loss:  11.9126
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.9092
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.918
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.9274
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff


Note

>>> s = 'bcdefghijklabcdefghijklabcdefghijklabcde'
>>> n = map(ord, s)
>>> chr(sum(n)/len(n))
'f'

• Welcome to the site! Can you post a plot of the test/train learning curves (loss vs. iterations)? – Emre May 15 '17 at 21:33
• I posted in an edit. I am more concerned about the lack of dynamics in than the loss. The predictions seem to converge to the average label – ignorance May 15 '17 at 21:55