The question is simple: is there any advantage in using sigmoid function in a convolutional neural network? Because every website that talks about CNN uses Relu function.


The reason that sigmoid functions are being replaced by rectified linear units, is because of the properties of their derivatives.

Let's take a quick look at the sigmoid function $\sigma$ which is defined as $\frac{1}{1+e^{-x}}$. The derivative of the sigmoid function is $$\sigma '(x) = \sigma(x)*(1-\sigma(x))$$ The range of the $\sigma$ function is between 0 and 1. The maximum of the $\sigma'$ derivative function is equal to $\frac{1}{4}$. Therefore when we have multiple stacked sigmoid layers, by the backprop derivative rules we get multiple multiplications of $\sigma'$. And as we stack more and more layers the maximum gradient decreases exponentially. This is commonly known as the vanishing gradient problem. The opposite problem is when the gradient is greater than 1, in which case the gradients explode toward infinity (exploding gradient problem).

Now let's check out the ReLU activation function which is defined as: $$R(x) = max(0,x)$$ The graph of which looks like


If you look at the derivatives of the function (slopes on the graph), the gradient is either 1 or 0. In this case we do not have the vanishing gradient problem or the exploding problem. And since the general trend in neural networks has been deeper and deeper architectures ReLU became the choice of activation.

Hope this helps.

| improve this answer | |
  • $\begingroup$ It's worth mentioning that ReLU is not differentiable at 0. Also ReLU is used because it's the simplest non-linearity to introduce between layers. $\endgroup$ – hh32 Dec 3 '16 at 11:49
  • $\begingroup$ So, what I think is that, when using ReLU, the learning rate should be much smaller than the learning rate when using sigmoid function, am I right? $\endgroup$ – Malvrok Dec 3 '16 at 13:27
  • $\begingroup$ It depends on a lot of things. e.g. architecture of the net, choice of layers, other hyper-parameters. $\endgroup$ – Armen Aghajanyan Dec 4 '16 at 6:54
  • $\begingroup$ this is a very clear explanation, however, it is still not clear to me why don't we encounter the vanishing gradient problem for the ReLU function when the gradient is 0.... can you comment on this? $\endgroup$ – jimijazz Apr 30 '18 at 14:39
  • $\begingroup$ The answer so far begs the question "Why not simply multiply the sigmoid activation function by 4?" The maximum gradient would then be 1, which is understood, in some sense, ideal. $\endgroup$ – Thomas Adkins Jun 14 '19 at 3:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.