In many ptb/mini shakespeare LSTM generator tutorials on web, people make input data, (where every batch is subsequent. for example, sequences in batch_2 are subsequent to sequences in batch_1, and batch_1 is fed right before batch_2) and feed those input data to dynamic_rnn. tutorial: https://r2rt.com/recurrent-neural-networks-in-tensorflow-ii.html
During training, these tutorials manually feed back the hidden state of LSTM, like codes below.
batch_size = 32
hidden_state_in = cell.zero_state(batch_size, tf.float32)
output, hidden_state_out = tf.nn.dynamic_rnn(cell, inputs,initial_state=hidden_state_in)
...
#For loop used in training: ...
output, hidden_state = sess.run([output, hidden_state_out],
feed_dict={hidden_state_in:hidden_state})
What makes me confused, is that when generating or testing ptb/shakespreare, these tutorials feed data, with batchsize==1 and varying seqlength (at training, batchsize was 32 or above).
def generate_characters(g, checkpoint, num_chars, prompt='A', pick_top_chars=None):
with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
g['saver'].restore(sess, checkpoint)
state = None
current_char = vocab_to_idx[prompt]
chars = [current_char]
for i in range(num_chars):
if state is not None:
feed_dict={g['x']: [[current_char]], g['init_state']: state}
#shape [BATCHSIZE, SEQLEN] with BATCHSIZE=1 and SEQLEN=1
else:
feed_dict={g['x']: [[current_char]]}
preds, state = sess.run([g['preds'],g['final_state']], feed_dict)
When using dynamic_rnn with LSTM, the hidden state has shape of [num_layers, 2, batch_size, state_size]. I just wonder how can we use data with different batch_size at training and testing. If hidden state is fed manually, what kind of trained feature is contained in LSTM graph? Is it cell_state, or the coefficients which are used in building states?
Also, when looking at generating text part of r2rt's tutorial above, first parts of generation seems weird. I guess this is because hidden_state (which is fed to dynamic_rnn's initial_state) at the beginning of generation is yet poorly configured, cause it's right after the first prompt. Am I right about this reason?