I work on a simple text generation problem using a portion of the Shakespeare dataset that I decided to use LSTM for. I primarily used this tutorial for reference. However, as I ran the below code, I noticed that the text generation section didn't work as expected: regardless of the input string (seed), the model always predicts the exact same word as having the highest probability. For example, when the input is just the one-word seed "i", the trained model would predict the next word to be "that". Feeding "i that" back to the model, it would again predict that the next word should be "that", and so forth, yielding a generated text of "i that that that that that..." which is obviously not how this model should be functioning.
Minimal reproducible code:
tokenizer = Tokenizer()
def get_sequence_of_tokens(corpus):
tokenizer.fit_on_texts(corpus)
total_words = len(tokenizer.word_index) + 1
input_sequences = []
for line in corpus:
token_list = tokenizer.texts_to_sequences([line])[0]
input_sequences.append(token_list)
return input_sequences, total_words
inp_sequences, total_words = get_sequence_of_tokens(train_data)
print("Total words:", total_words)
word_to_int = tokenizer.word_index
int_to_word = dict([(idx, word) for idx, word in enumerate(word_to_int)])
# Set maximum length
max_len = 25
# Pad all sequences
def generate_padded_sequences(input_sequences, max_len=max_len):
input_sequences = np.array(pad_sequences(input_sequences, padding='pre', maxlen=max_len+1))
predictors, label = input_sequences[:,:-1], input_sequences[:,-1]
labels = to_categorical(label, num_classes=total_words)
return predictors, labels
predictors, labels = generate_padded_sequences(inp_sequences)
def create_model(input_len, total_words):
model = Sequential()
model.add(Embedding(total_words, 32, input_length=input_len))
model.add(LSTM(64))
model.add(Dropout(0.1))
model.add(Dense(total_words, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')
return model
model_lstm = create_model(input_len=max_len, total_words=total_words)
# Train LSTM
history_lstm = model_lstm.fit(predictors, labels, epochs=5)
# Trying out text generation
seed_text = "i"
token_list = tokenizer.texts_to_sequences([seed_text])[0]
token_list = pad_sequences([token_list], maxlen=max_len, padding='pre')
print("Next word:", int_to_word[np.argmax(model_lstm.predict(token_list))]) # <-- this will always return same!
Imports and dataset:
import numpy as np
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense, Dropout, LSTM , Embedding, Flatten
from keras.preprocessing.sequence import pad_sequences
from keras.preprocessing.text import Tokenizer
from keras.utils.all_utils import to_categorical
train_data = ['o by no means honest ventidius i gave it freely ever and theres none can truly say he gives if our betters play at that game we must not dare to imitate them faults that are rich are fair', 'but was not this nigh shore', 'impairing henry strengthening misproud york the common people swarm like summer flies and whither fly the gnats but to the sun', 'what while you were there', 'chill pick your teeth zir come no matter vor your foins', 'thanks dear isabel', 'come prick me bullcalf till he roar again', 'go some of you knock at the abbeygate and bid the lady abbess come to me', 'an twere not as good deed as drink to break the pate on thee i am a very villain', 'beaufort it is thy sovereign speaks to thee', 'but say lucetta now we are alone wouldst thou then counsel me to fall in love', 'for being a bawd for being a bawd', 'all blest secrets all you unpublishd virtues of the earth spring with my tears', 'what likelihood', 'o find him', 'ay pilgrim lips that they must use in prayer', 'wouldst have me kneel', 'marry i fare well for here is cheer enough', 'that i may call thee something more than man and after that trust to thee', 'give me your hand']
