1
$\begingroup$

I know the concept between the vanishing and exploding gradient. I would like to know the possible causes of these phenomena. I think I read somewhere on the internet about the activation functions. Could someone please clarify? Any help will be greatly appreciated.

$\endgroup$

1 Answer 1

1
$\begingroup$

Gradient vanishing and exploding depend mostly on the following: too much multiplication in combination with too small values (gradient vanishing) or too large values (gradient exploding).

Activation functions are just one step in that multiplication when doing the backpropagation. If you have a good activation function, it could help in reducing these bad effects. One such activation function is ReLu. See this tutorial.

Other sources:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.