I am experimenting with a character-level LSTM model doing the standard task of predicting the next character given a sequence of characters. I am training the network using truncated backpropagation through time (BPTT).
I am trying to compare two techniques for setting up the training examples for this task. For my diagrams below, let's say the input sequence is the alphabet: "abcde..." and the horizon size for BPTT is 3. From what I understand, the standard way of training the network would be:
Training example 1 ('d' is the target character):
Training example 2:
Training example 3:
Technique #2:
Here is the alternative setup that seems reasonable to me:
Training example 1:
This training example has three separate target characters. The error from 'd' backpropagates through the full three time steps. However, error from 'c' backpropagates through two time steps and 'b' through one.
Training example 2:
Training example 3:
For training a single epoch over N characters, the second technique has a significant time complexity improvement. Increasing the horizon size has no impact on the running time. A larger horizon size will decrease the number of training examples, but each training example will have more target characters (i.e. more meaningful weight updates).
Is this second technique used at all for training LSTMs? Any relevant links? Why is the first technique more commonly used? Are there alternative training example setups which are also effective?
