I'm building an encoder-decoder neural network in Keras for sequence generation. The specific task is to try and change the styles of the text.

Both my encoder and decoders are LSTMs with latent dimensions of size 50. Also, the inputs to the network are one-hot encodings. I'm training the network using a couple of thousand data points and for over 100 epochs. The optimiser I use is rmsprop.

The accuracy and loss seem to decrease over time but during the inference stage, I find that every input sentence results in the same output. This is with a greedy sampling strategy.

I've played with different hyperparameters e.g. changing latent dims, batch-size, different optimisers etc. but to no avail. What strategies can I employ to assess whether this is a coding bug or an issue with the network?

  • $\begingroup$ Try to use leaky relu if you are using relu. $\endgroup$ Jul 26, 2018 at 16:46
  • $\begingroup$ I'm using the Keras defaults for LSTM which is tanh. For the final dense layer I just have a softmax function. $\endgroup$
    – Physbox
    Jul 26, 2018 at 18:17
  • $\begingroup$ How many instances do you have in your training set? $\endgroup$
    – JahKnows
    Jul 27, 2018 at 3:43
  • $\begingroup$ My training set has 2000 data points $\endgroup$
    – Physbox
    Jul 31, 2018 at 10:39


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.