By definition, Relu is max(0,f(x)). Then its gradient is defined as:
1 if x > 0 and 0 if x < 0.
Wouldn't this mean the gradient is always 0 (vanishes) when x < 0? Then why do we say Relu doesn't suffer from the gradient vanish problem?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
You're mostly correct! ReLU does have a problem with the gradient vanishing, but only on one side, so we call it something else: the 'dying ReLU problem'. See this stack overflow response for more information: What is the "dying ReLU" problem in neural networks?
It's a small semantic difference. Lots of functions (tanh and logistic/sigmoid) have derivatives very close to zero when you're outside the standard operating range. This is the 'vanishing gradient' issue. The worse you get, the harder it is to get back into the good zone. ReLU doesn't get worse the farther you are in the positive direction, so no vanishing gradient problem (on that side). This asymmetry might be enough to justify calling it something different, but the ideas are quite similar.
-
2$\begingroup$ Worth adding: The vanishing gradient problem tends to be about progressive changes over the depth of a network, and not directly about the properties of neuron transfer functions. $\endgroup$ May 25, 2016 at 18:55
$\begingroup$
$\endgroup$
Vanishing means that it goes towards 0 but will never really be 0. Having gradients of 0 makes for very easy calculations, having gradients close to 0 means that there are changes, just very tiny ones which mean slow learning and numerical issues. 1 and 0 are two of the easiest numbers to calculate in these kind of optimization problems.
answered May 4, 2016 at 18:40