I know the concept between the vanishing and exploding gradient. I would like to know the possible causes of these phenomena. I think I read somewhere on the internet about the activation functions. Could someone please clarify? Any help will be greatly appreciated.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Gradient vanishing and exploding depend mostly on the following: too much multiplication in combination with too small values (gradient vanishing) or too large values (gradient exploding).
Activation functions are just one step in that multiplication when doing the backpropagation. If you have a good activation function, it could help in reducing these bad effects. One such activation function is ReLu. See this tutorial.
Other sources:
A Gentle Introduction to Exploding Gradients in Neural Networks
Deep Learning book by Ian Goodfellow at el.: This book has a lot of examples with gradient problems (eg. Chapter 8.2.4: Cliffs and Exploding Gradients)
