# Algorithm for backpropagation through time

I am reading through this article trying to understand the bptt algorithm, in the context of an RNN. However there is one part I don’t understand:

layer_1_delta = (future_layer_1_delta.dot(synapse_h.T) +
layer_2_delta.dot(synapse_1.T)) +
sigmoid_output_to_derivative(layer_1)

# Located on line 99 in the article linked above


I can’t seem to figure out which mathematical formula this links to in the bptt algorithm to understand what it does, so could someone please provide it? Many thanks

• Do you have a link to the mathematical formulas to which you are referring? Commented Jul 21, 2018 at 11:29
• @n1k31t4 that’s the problem - I can’t seem to find one relating to the above line of code. I found some relating to the rest of the code however jramapuram.github.io/ramblings/rnn-backrpop Commented Jul 21, 2018 at 11:35
• @n1k31t4 forgot to tag you haha Commented Jul 21, 2018 at 12:11

As far as I can make out, this is simply the error aggregated at layer_1, meaning it is the sum of the errors from the current hidden layer, plus the error from all future hidden layers (in this case, that is only one layer: layer_2) plus the error from the output (prediction) layer - sigmoid_out.

I think that line of code (#99) is performing the loss update with respect to a chosen layer and all layers that are ahead in time - so closer to the network's output layer, but as we are moving backwards during backpropagation. It corresponds then, I believe, to equation 3.24 and 3.25 in Ilya Sutskever's PhD Thesis (on page 35).

So you are summing up the gradients of all layers ahead of the current layer, as well as the gradient of the output layer (the sigmoid). Equation 3.25 from the link above looks like this:

$$\frac{\delta L}{\delta W} = \sum_{t=1}^{T-1} \bigg( W'^T (r_{t+1} \odot (1 - r_{t+1}) \odot \frac{\delta L}{\delta r_{t+1}}) \bigg) v_t^T + \sum_{t=1}^{T-1} \frac{\delta log \hat P (v_t | r_{t-1})}{\delta W}$$

... where , where $v_t$ are the input variables and $r_t$ are the RNN’s hidden variables (all of which are deterministic). (from page 34)

The explanation given in the thesis:

The first summation in eq. 3.25 corresponds to the use of $W$ for computing $r_t$, and the second summation arises from the use of $W$ as RBM parameters for $log \hat P(v_t |r_{t−1})$, so each $∂log \hat P(v_{t+1}|r_t)/∂W$ term is computed with CD (contrastive divergence). Computing $∂L/∂rt$ is done most conveniently with a single backward pass through the sequence. It is also seen that the gradient of the RTRBM ( Recurrent Temporal Restricted Boltzmann Machine) would be computed exactly if CD were replaced with the derivatives of the RBM’s log probability.