3
$\begingroup$

I understand the concept of backpropagation in standard neural networks and backpropagation through time with RNNs, why this causes exponentially smaller gradients at earlier time steps and most of the maths behind it all, but what I don’t understand is why this affects the earlier timestep in particular? Since, the parameters (weights) in the RNN are all shared between timesteps, why is it that the earlier timestep is more affected? Wouldn’t they all be affected since they all share the same badly optimized weight which is never updated due to the many small terms in the multiplicative product which produces $\frac{\partial E}{\partial w} $? I feel like I'm totally misunderstanding something here. Many thanks

$\endgroup$
2
$\begingroup$

Of course that all weights are the same, but the update applied to the weights has a contribution from each of the timesteps, and the contribution associated with the first timesteps is what is more affected by the vanishing gradient problem.

$\endgroup$
6
  • $\begingroup$ Does this have an effect on all the parameters in the RNN, or just the weights of the edges between each time step? $\endgroup$
    – jim22394
    Apr 22 at 8:42
  • $\begingroup$ The gradient is propagated to update all parameters in the RNN, so the problem affects all parameters, and also any elements beyond the RNN, e.g. word embeddings. $\endgroup$
    – noe
    Apr 22 at 8:44
  • $\begingroup$ Thank you very much. Just to confirm my understanding, by contribution here, you mean the contribution to the sum which computes the partial derivate $\frac{\partial E}{\partial w} $? And this sum is formed based on the product rule right? $\endgroup$
    – jim22394
    Apr 22 at 8:47
  • $\begingroup$ Yes, that's right. $\endgroup$
    – noe
    Apr 22 at 9:04
  • $\begingroup$ Thanks again - So are all the partial derivatives for all the parameters in the RNN calculated in this way, as a sum of timestep contributions? $\endgroup$
    – jim22394
    Apr 22 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.