Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu has a disadvantage and it is its expected value. there is no limitation for the output of the Relu and so its expected value is not zero. I remember the time before the popularity of Relu that tanh was the most popular amongst machine learning experts rather than sigmoid. The reason was that the expected value of the tanh was equal to zero and and it helped learning in deeper layers to be more rapid in a neural net. Relu does not have this characteristic, but why it is working so good if we put its derivative advantage aside. Moreover, I guess the derivative also may be affected. Because the activations (output of Relu) are involved for calculating the update rules.

  • $\begingroup$ It is common to have some sort of normalization (e.g. batch normalization, layer normalization) together with ReLU. This adjusts the output range. $\endgroup$ – noe Oct 3 '17 at 14:22
  • $\begingroup$ @ncasas But in typical CNN normalizing the output of the relu is not common? At least I have never seen that. $\endgroup$ – Media Oct 3 '17 at 14:23
  • $\begingroup$ You are right, in not very deep CNNs it is normal not to have batch normalization. Have you considered the role of weight initial values? (e.g. He initialization) $\endgroup$ – noe Oct 3 '17 at 15:15
  • $\begingroup$ yes, actually they are for somehow preventing vanishing/exploding gradients, after some iterations the outputs get larger I guess. $\endgroup$ – Media Oct 3 '17 at 15:26

The biggest advantage of ReLu is indeed non-saturation of its gradient, which greatly accelerates the convergence of stochastic gradient descent compared to the sigmoid / tanh functions (paper by Krizhevsky et al).

But it's not the only advantage. Here is a discussion of sparsity effects of ReLu activations and induced regularization. Another nice property is that compared to tanh / sigmoid neurons that involve expensive operations (exponentials, etc.), the ReLU can be implemented by simply thresholding a matrix of activations at zero.

But I'm not convinced that great success of modern neural networks is due to ReLu alone. New initialization techniques, such as Xavier initialization, dropout and (later) batchnorm also played very important role. For example, famous AlexNet used ReLu and dropout.

So to answer your question: ReLu has very nice properties, though not ideal. But it truly proves itself when combined with other great techniques, which by the way solve non-zero-center problem that you've mentioned.

UPD: ReLu output is not zero-centered indeed and it does hurt the NN performance. But this particular issue can be tackled by other regularization techniques, e.g. batchnorm, which normalizes the signal before activation:

We add the BN transform immediately before the nonlinearity, by normalizing $x = Wu+ b$. ... normalizing it is likely to produce activations with a stable distribution.

  • 2
    $\begingroup$ I should've stressed this part: I was trying to say that ReLu alone doesn't solve this issue. You are right that ReLu output is not zero-centered and it does hurt the NN performance, unless the weights are regularized. But saturated gradients hurt the NN even more, so mass adoption of ReLu was a step forward despite its disadvantages. $\endgroup$ – Maxim Oct 3 '17 at 18:58
  • $\begingroup$ would you please say what do you mean by weights are regularized? in the answer and also the thing that you have emphasized. $\endgroup$ – Media Oct 3 '17 at 19:10
  • $\begingroup$ Updated my answer with some details about this particular issue $\endgroup$ – Maxim Oct 3 '17 at 19:24
  • $\begingroup$ What i find a bit confusing, why not just use the identity function? Whats the advantage of 0 for neg values? $\endgroup$ – Alex Dec 28 '17 at 9:42
  • 1
    $\begingroup$ 'flexibly squash the input through a function and pave the way recovering a higher-level abstraction structure', ok but why not use squared function for pos values? It will produce a far more intetesting derivative for example $\endgroup$ – Alex Dec 28 '17 at 9:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.