# Sigmoid vs Relu function in Convnets

The question is simple: is there any advantage in using sigmoid function in a convolutional neural network? Because every website that talks about CNN uses Relu function.

The reason that sigmoid functions are being replaced by rectified linear units, is because of the properties of their derivatives.

Let's take a quick look at the sigmoid function $\sigma$ which is defined as $\frac{1}{1+e^{-x}}$. The derivative of the sigmoid function is $$\sigma '(x) = \sigma(x)*(1-\sigma(x))$$ The range of the $\sigma$ function is between 0 and 1. The maximum of the $\sigma'$ derivative function is equal to $\frac{1}{4}$. Therefore when we have multiple stacked sigmoid layers, by the backprop derivative rules we get multiple multiplications of $\sigma'$. And as we stack more and more layers the maximum gradient decreases exponentially. This is commonly known as the vanishing gradient problem. The opposite problem is when the gradient is greater than 1, in which case the gradients explode toward infinity (exploding gradient problem).

Now let's check out the ReLU activation function which is defined as: $$R(x) = max(0,x)$$ The graph of which looks like

If you look at the derivatives of the function (slopes on the graph), the gradient is either 1 or 0. In this case we do not have the vanishing gradient problem or the exploding problem. And since the general trend in neural networks has been deeper and deeper architectures ReLU became the choice of activation.

Hope this helps.

• It's worth mentioning that ReLU is not differentiable at 0. Also ReLU is used because it's the simplest non-linearity to introduce between layers.
– hh32
Commented Dec 3, 2016 at 11:49
• So, what I think is that, when using ReLU, the learning rate should be much smaller than the learning rate when using sigmoid function, am I right? Commented Dec 3, 2016 at 13:27
• It depends on a lot of things. e.g. architecture of the net, choice of layers, other hyper-parameters. Commented Dec 4, 2016 at 6:54
• this is a very clear explanation, however, it is still not clear to me why don't we encounter the vanishing gradient problem for the ReLU function when the gradient is 0.... can you comment on this? Commented Apr 30, 2018 at 14:39
• The answer so far begs the question "Why not simply multiply the sigmoid activation function by 4?" The maximum gradient would then be 1, which is understood, in some sense, ideal. Commented Jun 14, 2019 at 3:08