# Exponential Linear Units (ELU) vs $log(1+e^x)$ as the activation functions of deep learning

It seems ELU (Exponential Linear Units) is used as an activation function for deep learning. But its' graph is very similar to the graph of $$log(1+e^x)$$. So why has $$log(1+e^x)$$ not been used as the activation functions instead of ELU?

In other words what is the advantage of ELU over $$log(1+e^x)$$?

## 1 Answer

ReLU and all its variants ( except ReLU-6 ) are linear i.e $$y = x$$ for values greater than or equal to 0.

This gives ReLU and specifically ELU an advantage like:

• Linearity means that the slope does not plateau or saturate when $$x$$ becomes larger. Hence, the vanishing gradient problem is solved.

Now, the graph $$y = log( 1 + e^x )$$ isn't linear for values > 0.

For larger negative values, the graph produces values which are very close to zero. This is also found in sigmoid where larger values produce a fully saturated activation. Hence, $$y = log( 1 + e^x )$$ can raise problems which sigmoid and tanh suffer.

About ELU:

ELU has a log curve for all negative values which is $$y = \alpha( e^x - 1 )$$. It does not produce a saturated firing for some extent but saturates for larger negative values.

See here for more information.

Hence, $$y = log( 1 + e^x )$$ is not used because of early saturation for negative values and also non linearity for values > 0. This may produce problems and even bring down some features which ReLU and variants exhibit.

• Thanks for your comments. But still I am not convinced. log(1+exp(x)), although it is non linear, it does not saturate for x>0. Do you say log(1+exp(x)) has early saturation for x<0 compared to ELU? – user570593 Jun 10 at 3:38
• For all values > 0, log(1+exp(x)) is a non-linear function whereas ReLU is linear ( y = x ). For large negative values, the value of log(1+exp(x)) becomes closer to zero, which indeed is a problem suffered by sigmoid too. – Shubham Panchal Jun 10 at 3:41