Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu has a disadvantage and it is its expected value. there is no limitation for the output of the Relu and so its expected value is not zero. I remember the time before the popularity of Relu that tanh was the most popular amongst machine learning experts rather than sigmoid. The reason was that the expected value of the tanh was equal to zero and and it helped learning in deeper layers to be more rapid in a neural net. Relu does not have this characteristic, but why it is working so good if we put its derivative advantage aside. Moreover, I guess the derivative also may be affected. Because the activations (output of Relu) are involved for calculating the update rules.
2 Answers
The biggest advantage of ReLu is indeed non-saturation of its gradient, which greatly accelerates the convergence of stochastic gradient descent compared to the sigmoid / tanh functions (paper by Krizhevsky et al).
But it's not the only advantage. Here is a discussion of sparsity effects of ReLu activations and induced regularization. Another nice property is that compared to tanh / sigmoid neurons that involve expensive operations (exponentials, etc.), the ReLU can be implemented by simply thresholding a matrix of activations at zero.
But I'm not convinced that great success of modern neural networks is due to ReLu alone. New initialization techniques, such as Xavier initialization, dropout and (later) batchnorm also played very important role. For example, famous AlexNet used ReLu and dropout.
So to answer your question: ReLu has very nice properties, though not ideal. But it truly proves itself when combined with other great techniques, which by the way solve non-zero-center problem that you've mentioned.
UPD: ReLu output is not zero-centered indeed and it does hurt the NN performance. But this particular issue can be tackled by other regularization techniques, e.g. batchnorm, which normalizes the signal before activation:
We add the BN transform immediately before the nonlinearity, by normalizing $x = Wu+ b$. ... normalizing it is likely to produce activations with a stable distribution.
-
2$\begingroup$ I should've stressed this part: I was trying to say that ReLu alone doesn't solve this issue. You are right that ReLu output is not zero-centered and it does hurt the NN performance, unless the weights are regularized. But saturated gradients hurt the NN even more, so mass adoption of ReLu was a step forward despite its disadvantages. $\endgroup$– MaximCommented Oct 3, 2017 at 18:58
-
$\begingroup$ would you please say what do you mean by weights are regularized? in the answer and also the thing that you have emphasized. $\endgroup$ Commented Oct 3, 2017 at 19:10
-
$\begingroup$ Updated my answer with some details about this particular issue $\endgroup$– MaximCommented Oct 3, 2017 at 19:24
-
$\begingroup$ What i find a bit confusing, why not just use the identity function? Whats the advantage of 0 for neg values? $\endgroup$– AlexCommented Dec 28, 2017 at 9:42
-
1$\begingroup$ 'flexibly squash the input through a function and pave the way recovering a higher-level abstraction structure', ok but why not use squared function for pos values? It will produce a far more intetesting derivative for example $\endgroup$– AlexCommented Dec 28, 2017 at 9:54
I think normalization issues are minor issues in numerical computations in general and machine learning in particular if you do not encounter significant numerical overflow or underflow.
The expected value issue is one of the normalization issues.
I know many people disagree with this claim. It is fine if you have the time to make all values close to 1 so that you get perfect numerical accuracy.
I usually use Relu and ignore all the normalization issues.
CNNnormalizing the output of thereluis not common? At least I have never seen that. $\endgroup$