I have trained RNN's before "by hand" using basic tools like Numpy or BLAS, but I am having trouble getting a simple RNN to converge in TF. Full Code

I tried standard things like adjusting the learning rate, the momentum, and adding noise to the gradient, but I am concerned that I don't understand what TF is doing underneath.

In a couple tutorials, it appeared that in TF you train the networks like Echo State Networks where the hidden state is evolved on random interconnections to train up a reservoir, then you use a linear (or non-linear) transformation to map the hidden state onto your labels.

I examined the gradients that Tensorflow uses to train, and it appears that it is training the internal state transition matrices as one would expect if using "Back-Propagation Through Time", which I can't tell if it is trying to use.

Can you help me understand what I am doing/understanding incorrectly with regards to training RNN's in TF?

cell_layer = tf.contrib.rnn.BasicLSTMCell(state_size, state_is_tuple=True)
cell = tf.contrib.rnn.MultiRNNCell([cell_layer]*1)
x, y_ = tf.placeholder(tf.float32, shape=(batch_size,1)), tf.placeholder(tf.float32, shape=(batch_size,1))
outputs, states = tf.nn.dynamic_rnn(cell, tf.expand_dims(x, -1), dtype=tf.float32)# init_state)

W = tf.Variable(tf.random_uniform((state_size, 1), -1.0, 1.0))
b = tf.Variable(tf.random_uniform((1, 1), -1.0, 1.0))
y = tf.matmul(tf.reshape(outputs, (-1, state_size)), W) + b

loss = tf.reduce_mean(tf.square(y - y_))
opt = tf.train.MomentumOptimizer(1e-3, 0.9)
grad = [(g + tf.random_uniform(v.shape, -1.0, 1.0) * tf.reduce_mean(tf.abs(v))*noise, v)
        for g, v in opt.compute_gradients(loss)]
train = opt.apply_gradients(grad)

EDIT: Loss plot

Loss vs. Iteration

I am less concerned about the loss than I am about the fact that there appear to not be any dynamics in between iterations. It appears to converge to the average. Below are some of the later losses and a part of the sequence to be predicted.

Loss:  11.9126
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.9092
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.918
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.9274
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff


>>> s = 'bcdefghijklabcdefghijklabcdefghijklabcde'
>>> n = map(ord, s)
>>> chr(sum(n)/len(n))
  • 1
    $\begingroup$ Welcome to the site! Can you post a plot of the test/train learning curves (loss vs. iterations)? $\endgroup$
    – Emre
    May 15, 2017 at 21:33
  • $\begingroup$ I posted in an edit. I am more concerned about the lack of dynamics in than the loss. The predictions seem to converge to the average label $\endgroup$
    – user32280
    May 15, 2017 at 21:55

1 Answer 1


I removed the gradient noise (which did not seem to help, at least as you did it) and replaced your momentum optimizer with Adam using the default hyperparameters and it just worked. After 10,000 epochs I got a loss of ~8 with bcdefghijklabcdefghijklabcdefghijklabcde (actual) vs ccdeeffgghhiccdeeffgghhiccdeeffgghhiccde (predicted).

The moral is that optimizing neural networks is still an art.

Plot every 100 iterations

p.s. It seems weird to use a regression loss with a classification problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.