Being faster or lower is a relative term and must be understood in the context of what it is comparing to. So, in order to understand this, we must first consider how gradient descent works with other types of the activation function.
Example Setup
Consider an MLP with $n$ hidden layers of size one.
$z_1 = W_1 x + b_1 $
$a_1 = f(z_1)$
...
$z_n = W_n a_{n-1} + b_n$
$y = f(z_n)$
where $f$ is the activation function.
Tanh and Sigmoid - Vanishing Gradient
Suppose $f$ is Tanh or Sigmoid activation function. The derivate of those functions are bounded between -1 to 1, or formally $f'(x) \in (-1, 1)$ for any $x$.
This causes a very important problem in deep learning known as "gradient vanishing problem". Let's consider the derivative of $y$ w.r.t $W_1$. By chain rule, we have
$$ \frac{df}{dW_1} = \frac{df}{dW_{n}} \frac{dW_{n}}{dW_{n-1}} ... \frac{dW_{2}}{dW_{1}}$$
and for any $0 < i < n$, notice that
$$ \frac{dX_{i}}{dX_{i-1}} = f'(W_{i-1}a_{i-2} + b_{i-1}) \times a_{i-2} \in (-1, 1)$$
(The first term is between $(-1, 1)$ because $f'$ is bounded as discussed earlier and $a_{i-2}$ is also between $(-1, 1)$ as squash the input value.)
So $\frac{df}{dW_1}$ is basically a product of lots of terms each is between (0, 1). The larger the $n$ (deeper the network) is the more of that term we need to multiply and as a result of the $\frac{df}{dW_1}$ becomes exponentially smaller.
Because of this exponential relationship, the gradient quickly becomes so small we can effectively consider it as zero. The consequence of having zero gradients is no learning can happen at all because our update rule for gradient descent is based on that gradient.
RELU and Dead Neuron
Relu is invented to deal with the vanishing gradient problem because its derivative is always 1 when $a_i > 0$ so when $f$ is RELU we have:
$$\frac{dX_{i}}{dX_{i-1}} = a_{i-2}$$
$$\frac{df}{dW_1} = a_1 a_2 a_3 ... a_{n-1}$$
It all nice and well when $x > 0$ but things fall apart whenever $x < 0$, this time not only is the gradient very close to zero, it is a pure zero. Once a neuron reaches there, there is not chance to get back at all. That's why this is known as the "Dead Neuron" problem
Leaky RELU and ELU
Leaky RELU and ELU is the natural development after RELU. They are similar to RELU as such the derivative equal to 1 when $x > 0$ but avoided "dead neuron" by avoiding zero derivates when $x<1$.
I quote the original paper for the difference between the two.
While LReLUs and PReLUs have negative values, too, they do not ensure
a noise-robust deactivation state. ELUs saturate to a negative value
with smaller inputs and thereby decrease the forward propagated
variation and information.
The intuitive explanation goes like the following. In ELU, whenever x became small enough, the gradient became really small and saturated (in the same way it happens for Tanh and Sigmoid). The small gradient means that the learning algorithm can focus on the tuning of other weights without worrying about the interactivity with the saturated neurons.
Consider a polynomial of degree 2 which can be represented as a smooth surface in a 3-d space. To find the local minimum, a gradient descent algorithm will need to consider the steepness in both x and y-direction. If the gradient is both negative in the x-direction and y-direction, it is not clear which way is better. So it is sensible to choose a path somewhere in between. But what if we already know everything is flat (zero gradients) in the x-direction, then it becomes a no-brainer to go for the y-direction. Or in other word, you search space become much smaller.
special note
In deep learning, there is a lot of claims without enough empirical evidence or in-depth understanding to support it. In ELU's case, while it might be true that it results in faster convergence for some datasets, it could also be true that it makes the learning algorithm to stuck at the local maximum for a different dataset. We just don't know enough yet.
