I am trying to train a RNN with text from wikipedia but I having having trouble getting the RNN to converge. I have tried increasing the batch size but it doesn't seem to be helping. All data is one hot encoded before being used and I am using the Adam optimizer which is implemented like this.

   for k in M.keys(): ##For k in weights
        M[k] = beta1 * M[k] + (1-beta1)*grad[k]
        R[k] = beta2 *R[k] + (1-beta2)*grad[k]**2
        m_k = M[k] / (1-beta1**n)
        r_k = R[k] / (1-beta2**n)
        model[k] = model[k] - alpha * m_k / np.sqrt(r_k + 1e-8)

Beta1 is set to 0.9, beta2 to 0.999 and alpha is set to 0.001. When I train it for 50,000 I get very high fluctuation of the cost and it never seems to significantly decrease (only sometimes due to the fluctuations (and I catch the weights with the lowest cost)). After sketching the cost of iterations I get a graph like this:enter image description here

It seems to be increasing on average only seeming to decrease to the the large fluctuations. What can I change to have better success and have it converge?

Thanks for any help


1 Answer 1


I think the problem lies with your text processing to one-hot vectors. Try using embeddings instead of one-hot vectors. An embedding is also a n-dimesional vector that allows words with similar meanings to have similar vectorial representation. One-hot vectors don't have such information. For them, there's a set of words of say cardinality c , then each vector is cx1 . The 1 in the vector just represents its location. It's like bit manipulation in this sense. So no semantic meaning is preserved. For e.g., in your corpus, there may be words like adore and love. Both have similar meaning. But one-hot vector for love and adore may be far away, depending upon where in set love and adore are mentioned.

But if you use embeddings, words with similar meanings will have similar representations in predefined vector space.

With this, your LSTM will learn dependencies better and will start converging.

Hope this helps :)

  • $\begingroup$ Thanks for the reply but I forgot to add that I am encoding individual characters rather than whole words, would embedding still help when using characters? $\endgroup$
    – treutm
    Dec 30, 2018 at 20:23
  • $\begingroup$ Yes, embeddings should perform better. $\endgroup$ Dec 31, 2018 at 3:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.