I now read a book titled "Hands-on Machine Learning with Scikit-Learn and TensorFlow" and on the chapter 11, it has the following description on the explanation of ELU (Exponential ReLU).

Third, the function is smooth everywhere, including around z = 0, which helps speed up Gradient Descent, since it does not bounce as much left and right of z = 0.

enter image description here

The z means the x-axis on the graph above. I understand the derivative is smooth since the z < 0 line has a curve and in that realm the derivative is no longer equal to 0.

However, why is it the case that if the function is "smooth everywhere, including around z=0", it speeds up Gradient Descent?

  • $\begingroup$ BTW am I in the correct SE? Data Science, Artificial Intelligence, and Cross Validated, (and also MathSE)... seems to me that many topics are highly related with each other... $\endgroup$
    – Blaszard
    Commented Aug 7, 2017 at 15:00
  • 2
    $\begingroup$ I think you would be safe posting this question either here on Data Science or on Cross Validated. Possibly it is ok for Artificial Intelligence too, but I am less familiar with that site. $\endgroup$ Commented Aug 7, 2017 at 17:53
  • $\begingroup$ Good question. Smoothness allows you to take bold steps in the right direction instead of tentatively taking baby steps because the gradient might change drastically at the next step. For a convergence analysis see e.g., Nesterov's Accelerated Gradient Descent for Smooth and Strongly Convex Optimization $\endgroup$
    – Emre
    Commented Sep 6, 2017 at 19:18
  • $\begingroup$ @NeilSlater AI has a very unclear scope. They close off such questions. So, DS and CV would be the best options :) $\endgroup$
    – Dawny33
    Commented Sep 7, 2017 at 6:19

4 Answers 4


A preliminary: there are three attributes of a function that are relevant here: continuous, monotonic, and differentiable. The RELU is continuous and monotonic nut not differentiable at z=0. The exponential relu or ELU is all three of those attributes.

The differential or gradient gives you a direction. When the derivative of a function is undefined at a point, then the direction of the gradient is indeterminate at that point.

When applying gradient descent, we wish to continuously modify parameters such that the loss function steadily decreases, which is the same as saying we wish to keep moving down towards minimum.

When the derivative of a loss function is undefined at some point, the gradient is indeterminate. This means the gradient descent could potentially move in the wrong direction. The magnitude of delay caused by this indeterminacy depends on the learning rate and other hyper-parameters. Regardless of the hyper-parameters, statistically, the undefined derivative in RELU at z=0, does contribute to slowing convergence of gradient descent.

  • $\begingroup$ It is very unlikely that the parameter becomes exactly z=0 after initialisation. $\endgroup$
    – Pieter
    Commented Dec 3, 2017 at 13:20

My guess it is due to the derivative, as ReLU has a discontinuous derivative at 0. Hence if you use the definition:

$$ f' \approx \frac{f(x+\epsilon) -f(x-\epsilon)}{2 \epsilon} $$

and $x$ is very close to 0, you will get many of those 'jumps'.


Being faster or lower is a relative term and must be understood in the context of what it is comparing to. So, in order to understand this, we must first consider how gradient descent works with other types of the activation function.

Example Setup

Consider an MLP with $n$ hidden layers of size one.

$z_1 = W_1 x + b_1 $

$a_1 = f(z_1)$


$z_n = W_n a_{n-1} + b_n$

$y = f(z_n)$

where $f$ is the activation function.

Tanh and Sigmoid - Vanishing Gradient

Suppose $f$ is Tanh or Sigmoid activation function. The derivate of those functions are bounded between -1 to 1, or formally $f'(x) \in (-1, 1)$ for any $x$.

This causes a very important problem in deep learning known as "gradient vanishing problem". Let's consider the derivative of $y$ w.r.t $W_1$. By chain rule, we have $$ \frac{df}{dW_1} = \frac{df}{dW_{n}} \frac{dW_{n}}{dW_{n-1}} ... \frac{dW_{2}}{dW_{1}}$$

and for any $0 < i < n$, notice that $$ \frac{dX_{i}}{dX_{i-1}} = f'(W_{i-1}a_{i-2} + b_{i-1}) \times a_{i-2} \in (-1, 1)$$

(The first term is between $(-1, 1)$ because $f'$ is bounded as discussed earlier and $a_{i-2}$ is also between $(-1, 1)$ as squash the input value.)

So $\frac{df}{dW_1}$ is basically a product of lots of terms each is between (0, 1). The larger the $n$ (deeper the network) is the more of that term we need to multiply and as a result of the $\frac{df}{dW_1}$ becomes exponentially smaller. Because of this exponential relationship, the gradient quickly becomes so small we can effectively consider it as zero. The consequence of having zero gradients is no learning can happen at all because our update rule for gradient descent is based on that gradient.

RELU and Dead Neuron

Relu is invented to deal with the vanishing gradient problem because its derivative is always 1 when $a_i > 0$ so when $f$ is RELU we have: $$\frac{dX_{i}}{dX_{i-1}} = a_{i-2}$$ $$\frac{df}{dW_1} = a_1 a_2 a_3 ... a_{n-1}$$

It all nice and well when $x > 0$ but things fall apart whenever $x < 0$, this time not only is the gradient very close to zero, it is a pure zero. Once a neuron reaches there, there is not chance to get back at all. That's why this is known as the "Dead Neuron" problem

Leaky RELU and ELU

Leaky RELU and ELU is the natural development after RELU. They are similar to RELU as such the derivative equal to 1 when $x > 0$ but avoided "dead neuron" by avoiding zero derivates when $x<1$.

I quote the original paper for the difference between the two.

While LReLUs and PReLUs have negative values, too, they do not ensure a noise-robust deactivation state. ELUs saturate to a negative value with smaller inputs and thereby decrease the forward propagated variation and information.

The intuitive explanation goes like the following. In ELU, whenever x became small enough, the gradient became really small and saturated (in the same way it happens for Tanh and Sigmoid). The small gradient means that the learning algorithm can focus on the tuning of other weights without worrying about the interactivity with the saturated neurons.

Consider a polynomial of degree 2 which can be represented as a smooth surface in a 3-d space. To find the local minimum, a gradient descent algorithm will need to consider the steepness in both x and y-direction. If the gradient is both negative in the x-direction and y-direction, it is not clear which way is better. So it is sensible to choose a path somewhere in between. But what if we already know everything is flat (zero gradients) in the x-direction, then it becomes a no-brainer to go for the y-direction. Or in other word, you search space become much smaller.

special note

In deep learning, there is a lot of claims without enough empirical evidence or in-depth understanding to support it. In ELU's case, while it might be true that it results in faster convergence for some datasets, it could also be true that it makes the learning algorithm to stuck at the local maximum for a different dataset. We just don't know enough yet.


I have an intuitive notion of why smooth functions are faster to optimize but no mathematical proof or anything.

Gradient descent computes the derivative of the activation function to determine the change in weights. When the activation function has a hard cut (e.g. at z=0 for ReLu) the activation of the unit can changes radically (i.e. always zero or linear) for specific data points when changing the weights.

The other weights need to adept to this radically different behaviour of a specific unit for specific data points. If, however, the behaviour of the unit change radically again during the next epoch the network keeps on adepting towards changes in the previous epoch.

With a smooth function there are no such radical changes. And thus the network can more gradually stabilise.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.