In many descriptions of a CNN i often read that at the end of the Convolutional layer, a ReLU function is needed, for two reasons: first it solves many problems about the vanishing gradient problem, second it enables the network to learn more complex feature in the data. What i cannot figure out is, how is it possible to understand such an improvement by just looking at the form of the ReLU function. I removes all negative values and this translates in getting rid of things such as smooth transitions of grey in an image. But starting from this i do not see how this should lead to the capability of detecting more complex features in an input image.


1 Answer 1


Before ReLU became popular, the usual activation functions were tanh and sigmoid. These activation functions suffer a problem called "vanishing gradient", which caused the gradient to be very small in the initial layers because it was "squeezed" by the regions of those functions that were far from the origin. It mainly happened when you stacked a lot of layers and therefore the gradient was squeezed once and again in backpropagation. The effect of this problem is that the network learned very very slowly and even failed to learn at all.

ReLU does not suffer from the vanishing gradient problem, as it does not squeeze the gradient. Therefore, you can stack more layers which, in the end, allows you to recognize more complex stuff.

Nevertheless, ReLU have their own problems, like the "Dying ReLU" problem, in which the optimization process gets "trapped" in the area where the ReLU is zero. That's why variants like Leaky ReLU were proposed; they don't output zero but a small slope, preventing the dying ReLU issue.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.