I'm building an encoder-decoder neural network in Keras for sequence generation. The specific task is to try and change the styles of the text.
Both my encoder and decoders are LSTMs with latent dimensions of size 50. Also, the inputs to the network are one-hot encodings. I'm training the network using a couple of thousand data points and for over 100 epochs. The optimiser I use is rmsprop.
The accuracy and loss seem to decrease over time but during the inference stage, I find that every input sentence results in the same output. This is with a greedy sampling strategy.
I've played with different hyperparameters e.g. changing latent dims, batch-size, different optimisers etc. but to no avail. What strategies can I employ to assess whether this is a coding bug or an issue with the network?