6
$\begingroup$

Vanishing gtadient causes too small values which are come from non-linear functions like sigmoid, tanh. LSTM architecture was developed to fight these issues. But how can it avoid a problem if there are 3 sigmoid, 2 tanh functions and, moreover, they are combinated via multiplication?

$\endgroup$
1
1
$\begingroup$

There are two factors that affect the magnitude of gradients - the weights and the activation functions (or more precisely, their derivatives) that the gradient passes through.

If either of these factors is smaller than 1, then the gradients may vanish in time; if larger than 1, then exploding might happen. For example, the tanh derivative is <1 for all inputs except 0; sigmoid is even worse and is always ≤0.25.

In the recurrency of the LSTM the activation function is the identity function with a derivative of 1.0. So, the backpropagated gradient neither vanishes or explodes when passing through, but remains constant. The effective weight of the recurrency is equal to the forget gate activation. So, if the forget gate is on (activation close to 1.0), then the gradient does not vanish. Since the forget gate activation is never >1.0, the gradient can't explode either. So that's why LSTM is so good at learning long range dependencies.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.