I am trying to create a word-level Haiku generator using an LSTM neural network. I am scraping haikus from Reddit's r/haiku, and wanted to start with a "simple" model: my training data is the set of all haikus, flattened, and split into trigrams, such that the first two words of the trigram are the features and the last word is the label. I chose a "context size" of two just because I wanted to start with a small size, considering how small the haikus themselves are. So this is essentially a text prediction task with a small context window.

E.g the first couple (X,y) pairs (There are ~ 90,000 such pairs) look like:

[('delicate savage', '/'),
 ('savage /', "you'll"),
 (" / you'll", 'never'),
 ("you'll never", 'hold')... ]

(I am using the '/' character to designate a new stanza, and I also added '$' to indicate the end of the haiku, to see if my model will learn such distinctions present in a haiku's format)

I then encode my sequences using an index mapping and feed this into an embedding layer when I train my model.

The model:

    Embedding(input_dim=vocab_size, output_dim=60, input_length=seq_length), #seq length is 2
    Dense(100,kernel_regularizer=l2(0.01),activation = 'relu'),
    Dense(vocab_size, activation = 'softmax')

My performance is pretty bad: about 45% training accuracy and 15% validation accuracy. When I generate haiku sequences using a fit model, the model is often clueless and returns None.

I recognize that there are several places my model could be going wrong, but am really not sure where. I've tried various regularization techniques (dropout, L2) and modifying my neural network's architecture (# of layers and nodes) but without any luck. Before I try using pre-trained embedding, what can I do to improve my ability to predict text with my data?

If it helps, here's my notebook, using a language model heavily based on this implementation.

  • $\begingroup$ Perhaps you can also try using current state of the art Transformer based models? $\endgroup$ Commented May 29, 2021 at 5:08
  • $\begingroup$ State of the art models will indeed be looked into, but considering my scores at the moment - in the context of general language models - I think I should focus on my hyperparametrization and data structure and not jump the gun. $\endgroup$ Commented Jun 1, 2021 at 1:46

1 Answer 1


I don't think you actually need regularization since regularization techniques are needed in an overfitted model but your model works poorly on both train/validation sets, which is evidence of underfitting.

If you want to improve your network, you can consider adding more layers to your network (e.g. stacked LSTM). I don't know what exactly the Haiku is but if the Haiku needs a specific rule for either rhythm or rhyme, you may consider taking input as character-level like ELMo.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.