# LSTM - divide gradients by number of timesteps IMMEDIATELY or in the end?

From this answer I know that

the gradient of an average of many functions, is equal to the average of the gradients of those functions taken separately.

The error gradient that you want to calculate for gradient descent is $$\nabla_{\theta} C(X, \theta)$$, which you can therefore write as:

$$\nabla_{\theta} C(X, \theta) = \nabla_{\theta}(\frac{1}{|X|}\sum_{x \in X} L(x, \theta))$$

The derivative of the sum of any two functions is the sum of the derivatives, i.e.

$$\frac{d}{dx}(y+z) = \frac{dy}{dx} + \frac{dz}{dx}$$

In addition, any fixed multiplier that doesn't depend on the parameters you are taking the gradient with (in this case, the size of the dataset) can just be treated as an external factor:

$$\nabla_{\theta} C(X, \theta) = \frac{1}{|X|}\sum_{x \in X} \nabla_{\theta} L(x, \theta)$$

Question:

When working with LSTM specifically, am I indeed allowed to apply this $$\frac{1}{|X|}$$ at the end of the backprop once I've summed-up gradients of weights across all timesteps?

Or should I always apply $$\frac{1}{|X|}$$ straight away, to any gradient flowing into the top layers of my network?

This bothers me because LSTM is not merely multiplying things like a basic RNN would. Instead, LSTM has summation, which then gets multiplied with other things as we descend to earlier timesteps.

For example, this happens when we have to add some gradient flowing from all the 4 gates the LSTM into grad_wrt_resultAtPreviosTimesep (It will be required to compute grad during previous timestep once we get there)

Using an overly-simplified algebra example:

I feel that multiplying immediately by $$\frac{1}{X}$$ would represent the left side of this expression: $$(\frac{1}{X}a + 40)*100 \neq \frac{1}{X} (a+40)*100$$. On the other hand, multiplying by $$\frac{1}{X}$$ in the end of backprop would represent the right side.

Edit:

Just as a reminder, here is what LSTM looks like at each timestep, taken from this blog:

• I think you are confusing forward and backward pass. "apply 1/|X| straight away, to any gradient flowing into the top layers of my network?" sounds like a forward pass. "at the end of the backprop once I've summed-up gradients of weights across all timesteps?" sounds like backward pass. Commented Nov 25, 2018 at 14:34
• Here I am only considering backward pass - I imagine backward pass as a top-down process and forward pass as bottom-up
– Kari
Commented Nov 26, 2018 at 11:57

Answer to your question is it doesn't matter. The gradient is just a product of Jacobians (because of the chain rule), so it doesn't matter if you multiply the result or the intermediate multiplicand, these give the same result. Or, better yet, is to multiply the function you're differentiating, since $$\tfrac{d(\alpha f(x))}{dx} = \alpha f'(x)$$. So instead of manipulating the gradients you should just average your loss over whatever you want it to be averaged over.

Note that I didn't assume anything about the function above. It's because these are fairly general basic rules, and they apply to all kinds of functions, RNNs and LSTMs included.

That said, you might not want to actually average your loss (remembers, it's the same as averaging the gradients) over the timesteps. Average over inputs x – yeah, sure. Over timesteps? Not necessarily a good idea.

If you do average over timesteps, then you essentially decrease importance of longer sequences, and thus your model will care less of making more mistakes in them, whereas if you do not average over timesteps (or if they are all of the same length), then every timestep-loss has the same weight, and the model cares about all of them equally.

• Thank you @ArtemSobolev. I see, so it can be done either way because of the chain rule - these functions are netsted in one another, so any averaging can be slapped-on afterwards. "Average over inputs x" - do you mean to average over the number of examples? (examples of a minibatch).
– Kari
Commented Nov 26, 2018 at 13:07
• "So instead of manipulating the gradients you should just average your loss over whatever you want it to be averaged over" - Let's say I want to average my loss by $\frac{1}{m}$ where $m$ is the number of examples (number of sequences) in my minibatch. I think in that case, wouldn't I still have to manipulate the gradients by this $\frac{1}{m}$ anyway, because it's part of my Loss function. Could you please edit/provide example?
– Kari
Commented Nov 26, 2018 at 13:16
• In other words, my chain rule would force me to descend downwards through the $\frac{1}{m}$ or any similar coefficient (were I to use one), and then through the sums of losses. Only then can my gradient enter the layers of the network.
– Kari
Commented Nov 26, 2018 at 13:20
• @Kari, Right, essentially you have C(θ) = 1/m S(θ) where S(θ) = ∑ᵢ L(xᵢ, θ), so you first differentiate C as dC(θ)/dθ = 1/m dS(θ)/dθ, so if you use automatic differentiation, differentiating average loss C(θ) averages the gradients for you. Commented Nov 27, 2018 at 20:00