I'm trying to create a neural network that can learn how to write text character by character from the book David Copperfield (via Project Gutenburg).
It starts great, then forgets punctuation around epoch 25 and devolves into nonsense at epoch 26. I've been trying to find a starting point where I can start attacking this problem. I've read research papers on the concept of clipping the gradient to stop it from vanishing or exploding, but I'm having a hard time finding a way to first visualize the gradient and what's going wrong and then how to clip it at the most appropriate value.
I've saved checkpoint models from all fifty epochs.
I've tried using a clipping the gradient at 5 based on a research paper about gradient clipping in LSTMs and it didn't change anything. I don't have to budget to find the optimum value through experimentation, but if that's the only way to do it I'll make it work.
I've been working on this a long time, but I'm self-taught and feel a a bit out of my depth here. A nudge in the right direction from a subject matter expert would be greatly appreciated.
Epoch 20:
and that I saw Traddles the service in the house, person a sort of something and so great hand of the counter being a prospect of Mr. Omer, had replied to go and so moment as a case so refersing the house of his man as distracting him on the door and last hearing her after for the streng to a head, and Mr. Micawber, that many beer and when I was so consequent when the man. I can be so looked to keep me all the black desiring the constinging and enearly to his part of more arms were in the scound as my delight, that she asked the time and herself of what I had been with the windows years of such a green eye of the most night was a dear, but that there are you? I began to make dear, she was not that a present words, long a back hand with a better was many to a relief in search had one been what he was to say that the hand sitting weeks, and made with a present time, and my face, so he sat no more about the mount, and set of the time of petitution and the door was in a proceeding the spirit,
Epoch 25:
sir, I am willing to the beautiful thought who was not her to the best flew for a son, when I said, said I. When she had to speak the strong spirits of this comprier of an and bill time of the resist to my street at the garden with it, and that he before my for the sense of the room in the old evening in the time with a real arms child would take it a little books to the send with a shining the care that I cant have seen it a good shoulder of the morning for a moment in a time of a good mist from the expect of laughing so like a deport for one of the carrier of that success of a spoke with the wild of the table just over and thing being pretending in the proposition that there took a far in my devoted towards me what he went to a sense of the stare to the breditol for the great contriout, and that I were done in the shadow and man of our fathers dream at once with me, as to say, but he said a man. I think it was one far that he had been over the desire that was not loved in the grief that could says more
Epoch 26:
spi , i ed o eoa t i i. nhsaae s t?h h t h n
t cha,p t r ieto t wo a hw ne s uawpai ,y na a e t dttte t?atsh oh h i au a a a ddn e haf e s t.rooe wt etdt s
t a ,, i t e a h e a, dt os rhr tis m elrii e a ao ty otatp ya r t a ty o he, ee , s s i. hn e o te a o o se s h u te senea e tt s d ew , ie s I ee ihi hnts a an r rv otso t a eshne tta o tt
e o m l h arnt led n sa a a e h tww n ee t h ha ,tdeh t e nntt i atnr,e wt eee hb a t n oea a e ei t -. de e a e s e n atehh h e a s ef e n to hr d , eh In.i o st watn t htih e io tt a x h s s , e s r ohsmtldal er e
n, t,dtthan,t h s hdhe oa oh tbh t ot isn o e tr et ttnm an a ot ng c a ds tr .s t h a t t ewehso n sr o e se i e e httst b i tt . t n I he o w so tt t,w sttt nta t ai o
Code Follows: (I started with Tensorflow, but switched to TFLearn for simplicity. I'm willing to learn any frameworks that have the tools to solve the problem, though. I'm a self-taught student, so learning is really the only objective here.)
import time
start_script_time = time.time()
import numpy as np
import tflearn
import random
import pickle
''' create data '''
log_file = 'dickens_log.txt'
def my_log(text, filename=log_file):
text = str(text)
print(text)
with open(filename, 'a', newline='\n') as file:
file.write(text + '\n')
try:
book_name = 'as_loaded.txt'
book = open(book_name, errors='ignore',
encoding='ascii', newline='\n').read()
except:
book_name = 'copperfield.txt'
book = open(book_name, errors='ignore',
encoding='utf-8', newline='\n').read()
#book = book.replace('\r', '')
#book = book.replace('\n', ' ')
with open('as_loaded.txt', 'w', newline='\n') as file:
file.write(book)
# make smaller slice for quickly testing code on CPU
# book = book[0:1500]
# del(book_name)
# length of strings in the training set
string_length = 30
def process_book(book, string_length, redundant_step=3):
# Remember to pickle to dictionary as a binary. This is pretty critical for loading your model on a different machine than you trained on.
try:
pickle_ld = open('charDict.pi', 'rb')
charDict = pickle.load(pickle_ld)
pickle_ld.close()
except:
# dictionary of character-number pairs
chars = sorted(list(set(book)))
charDict = dict((c, i) for i, c in enumerate(chars))
#charDict.pop('\r')
pickle_sv = open('charDict.pi', 'wb')
pickle.dump(charDict, pickle_sv)
pickle_sv.close()
len_chars = len(charDict)
# train is a string input and target is the
# expected next character
train = []
target = []
for i in range(0, len(book)-string_length, redundant_step):
train.append(book[i:i+string_length])
target.append(book[i+string_length])
# create containers for data with appropriate dimensions
# 3D (n_samples, sample_size, n_categories)
X = np.zeros((len(train), string_length, len_chars), dtype=np.bool)
# 2D (n_samples, n_categories)
y = np.zeros((len(train), len_chars), dtype=np.bool)
# fill arrays
for i, string in enumerate(train):
for j, char in enumerate(string):
# X is a sparse 3D tensor where a 1 value signals
# that a information is present in 3rd dimension index
X[i, j, charDict[char]] = 1
y[i, charDict[target[i]]] = 1
return charDict, X, y
charDict, X, y = process_book(book, string_length)
''' build the network '''
# number of hidden layers in each LSTM layer
lstm_hidden = 512
drop_rate = 0.5
net = tflearn.input_data(shape=(None, string_length, len(charDict)))
# input shape is the length of the strings by the number of characters
# leading None is necessary if no placeholders
net = tflearn.lstm(net, lstm_hidden, return_seq=True)
net = tflearn.dropout(net, drop_rate)
# You have to use a separate dropout layer. There's a glitch where tflean
# will drop out all the time, not just during training, making prediction
# impossible.
net = tflearn.lstm(net, lstm_hidden, return_seq=True)
net = tflearn.dropout(net, drop_rate)
net = tflearn.lstm(net, lstm_hidden, return_seq=False)
net = tflearn.dropout(net, drop_rate)
net = tflearn.fully_connected(net, len(charDict), activation='softmax')
net = tflearn.regression(net, optimizer='adam',
loss='categorical_crossentropy',
learning_rate=0.005)
# https://www.quora.com/What-is-gradient-clipping-and-why-is-it-necessary
model = tflearn.SequenceGenerator(net, dictionary=charDict,
seq_maxlen=string_length,
clip_gradients=5,
checkpoint_path='model_checkpoint_v3')
my_log('Character dictionary for ' + book_name)
my_log(charDict)
my_log('charDict length: ' + str(len(charDict)))
my_log('&&&&&&&&&&&&&&&&&')
def random_seed_test(book, temp=0.5, gen_length=300):
my_log('#######################')
seed_no = random.randint(0, len(book) - string_length)
seed = book[seed_no : seed_no + string_length]
my_log('(temp ' + str(temp) + ') ' + 'Seed: "' + seed + '"')
my_log('++++++++++++++++++++++')
my_log(model.generate(seq_length=gen_length, temperature=temp,
seq_seed=seed))
my_log('#######################')
# If you train one epoch at a time in a loop, you can get an idea
# of how the model progressed. With other ML problems, error rate and
# accuracy reveal a lot, but with this problem performance is subjective.
for epoch in range(50):
start_epoch = time.time()
my_log('======================================================')
my_log('Begin epoch %d' % (epoch+1))
model.fit(X, y, validation_set=0.1, batch_size=128, n_epoch=1)
my_log('End epoch %d' % (epoch+1))
epoch_time = time.time() - start_epoch
my_log('This epoch took ' + str(epoch_time) + ' seconds.')
random_seed_test(book, temp=0.5, gen_length=1000)
random_seed_test(book, temp=0.75, gen_length=1000)
random_seed_test(book, temp=1.0, gen_length=1000)
my_log('End epoch %d' % (epoch+1))
my_log('======================================================')
full_time = time.time() - start_script_time
my_log('This program took ' + str(full_time) + ' seconds.')
model.save('dickens_compute_4.model')
my_log('finished')
