Vanishing gtadient causes too small values which are come from non-linear functions like sigmoid, tanh. LSTM architecture was developed to fight these issues. But how can it avoid a problem if there are 3 sigmoid, 2 tanh functions and, moreover, they are combinated via multiplication?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ How does LSTM prevent the vanishing gradient problem? $\endgroup$– Franck DernoncourtMar 7, 2017 at 21:33
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
There are two factors that affect the magnitude of gradients - the weights and the activation functions (or more precisely, their derivatives) that the gradient passes through.
If either of these factors is smaller than 1, then the gradients may vanish in time; if larger than 1, then exploding might happen. For example, the tanh derivative is <1 for all inputs except 0; sigmoid is even worse and is always ≤0.25.
In the recurrency of the LSTM the activation function is the identity function with a derivative of 1.0. So, the backpropagated gradient neither vanishes or explodes when passing through, but remains constant.
The effective weight of the recurrency is equal to the forget gate activation. So, if the forget gate is on (activation close to 1.0), then the gradient does not vanish. Since the forget gate activation is never >1.0, the gradient can't explode either.
So that's why LSTM is so good at learning long range dependencies.
