2
$\begingroup$

I am experimenting with a character-level LSTM model doing the standard task of predicting the next character given a sequence of characters. I am training the network using truncated backpropagation through time (BPTT).

I am trying to compare two techniques for setting up the training examples for this task. For my diagrams below, let's say the input sequence is the alphabet: "abcde..." and the horizon size for BPTT is 3. From what I understand, the standard way of training the network would be:

Training example 1 ('d' is the target character):

enter image description here

Training example 2:

enter image description here

Training example 3:

enter image description here

Technique #2:

Here is the alternative setup that seems reasonable to me:

Training example 1:

enter image description here

This training example has three separate target characters. The error from 'd' backpropagates through the full three time steps. However, error from 'c' backpropagates through two time steps and 'b' through one.

Training example 2:

enter image description here

Training example 3:

enter image description here

For training a single epoch over N characters, the second technique has a significant time complexity improvement. Increasing the horizon size has no impact on the running time. A larger horizon size will decrease the number of training examples, but each training example will have more target characters (i.e. more meaningful weight updates).

Is this second technique used at all for training LSTMs? Any relevant links? Why is the first technique more commonly used? Are there alternative training example setups which are also effective?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.