Feeding back hidden state manually in tf.nn.dynamic_rnn (Tensorflow)

Question

In many ptb/mini shakespeare LSTM generator tutorials on web, people make input data, (where every batch is subsequent. for example, sequences in batch_2 are subsequent to sequences in batch_1, and batch_1 is fed right before batch_2) and feed those input data to dynamic_rnn. tutorial: https://r2rt.com/recurrent-neural-networks-in-tensorflow-ii.html

During training, these tutorials manually feed back the hidden state of LSTM, like codes below.

batch_size = 32
hidden_state_in = cell.zero_state(batch_size, tf.float32) 
output, hidden_state_out = tf.nn.dynamic_rnn(cell, inputs,initial_state=hidden_state_in)
...
#For loop used in training: ...
    output, hidden_state = sess.run([output, hidden_state_out],
                            feed_dict={hidden_state_in:hidden_state})

What makes me confused, is that when generating or testing ptb/shakespreare, these tutorials feed data, with batchsize==1 and varying seqlength (at training, batchsize was 32 or above).

def generate_characters(g, checkpoint, num_chars, prompt='A', pick_top_chars=None):
    with tf.Session() as sess:
        sess.run(tf.initialize_all_variables())
        g['saver'].restore(sess, checkpoint)
        state = None
        current_char = vocab_to_idx[prompt] 
        chars = [current_char]

        for i in range(num_chars):
             if state is not None:
                feed_dict={g['x']: [[current_char]], g['init_state']: state}
                #shape [BATCHSIZE, SEQLEN] with BATCHSIZE=1 and SEQLEN=1 
            else:
                feed_dict={g['x']: [[current_char]]}
            preds, state = sess.run([g['preds'],g['final_state']], feed_dict)

When using dynamic_rnn with LSTM, the hidden state has shape of [num_layers, 2, batch_size, state_size]. I just wonder how can we use data with different batch_size at training and testing. If hidden state is fed manually, what kind of trained feature is contained in LSTM graph? Is it cell_state, or the coefficients which are used in building states?

Also, when looking at generating text part of r2rt's tutorial above, first parts of generation seems weird. I guess this is because hidden_state (which is fed to dynamic_rnn's initial_state) at the beginning of generation is yet poorly configured, cause it's right after the first prompt. Am I right about this reason?

Answer 1

There's a lot going on inside an LSTM, so it's easy to get confused. I think you are confusing the state and the weights:

• The weights are trained at training time and are not updated during prediction. This is like the weights of any other neural network.
• The state updates as the model moved "forward" through the text. It is essentially what enables the LSTM to keep track of where it currently is in a sequence.

The weights have a fixed size, of course, but when generating text the LSTM acts only on the last character and the current state, which is why the sequence length is set to 1 (and if you're only generating one text at a time, the batch length is also 1. you could increase the batch size to generate multiple texts at the same time).