# Why is my neural language model performing so poorly?

I am trying to create a word-level Haiku generator using an LSTM neural network. I am scraping haikus from Reddit's r/haiku, and wanted to start with a "simple" model: my training data is the set of all haikus, flattened, and split into trigrams, such that the first two words of the trigram are the features and the last word is the label. I chose a "context size" of two just because I wanted to start with a small size, considering how small the haikus themselves are. So this is essentially a text prediction task with a small context window.

E.g the first couple (X,y) pairs (There are ~ 90,000 such pairs) look like:

[('delicate savage', '/'),
('savage /', "you'll"),
(" / you'll", 'never'),
("you'll never", 'hold')... ]


(I am using the '/' character to designate a new stanza, and I also added '\$' to indicate the end of the haiku, to see if my model will learn such distinctions present in a haiku's format)

I then encode my sequences using an index mapping and feed this into an embedding layer when I train my model.

The model:

Sequential([
Embedding(input_dim=vocab_size, output_dim=60, input_length=seq_length), #seq length is 2
LSTM(100,return_sequences=True),
LSTM(100),
Dense(100,kernel_regularizer=l2(0.01),activation = 'relu'),
Dense(vocab_size, activation = 'softmax')
])


My performance is pretty bad: about 45% training accuracy and 15% validation accuracy. When I generate haiku sequences using a fit model, the model is often clueless and returns None.

I recognize that there are several places my model could be going wrong, but am really not sure where. I've tried various regularization techniques (dropout, L2) and modifying my neural network's architecture (# of layers and nodes) but without any luck. Before I try using pre-trained embedding, what can I do to improve my ability to predict text with my data?

If it helps, here's my notebook, using a language model heavily based on this implementation.

• Perhaps you can also try using current state of the art Transformer based models? – Devashish Prasad May 29 at 5:08
• State of the art models will indeed be looked into, but considering my scores at the moment - in the context of general language models - I think I should focus on my hyperparametrization and data structure and not jump the gun. – Brian Barry Jun 1 at 1:46