5
$\begingroup$

It seems ELU (Exponential Linear Units) is used as an activation function for deep learning. But its' graph is very similar to the graph of $log(1+e^x)$. So why has $log(1+e^x)$ not been used as the activation functions instead of ELU?

In other words what is the advantage of ELU over $log(1+e^x)$?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
3
$\begingroup$

ReLU and all its variants ( except ReLU-6 ) are linear i.e $ y = x $ for values greater than or equal to 0.

This gives ReLU and specifically ELU an advantage like:

  • Linearity means that the slope does not plateau or saturate when $x$ becomes larger. Hence, the vanishing gradient problem is solved.

enter image description here

Now, the graph $ y = log( 1 + e^x ) $ isn't linear for values > 0.

For larger negative values, the graph produces values which are very close to zero. This is also found in sigmoid where larger values produce a fully saturated activation. Hence, $ y = log( 1 + e^x ) $ can raise problems which sigmoid and tanh suffer.

About ELU:

enter image description here

ELU has a log curve for all negative values which is $ y = \alpha( e^x - 1 )$. It does not produce a saturated firing for some extent but saturates for larger negative values.

See here for more information.

Hence, $ y = log( 1 + e^x ) $ is not used because of early saturation for negative values and also non linearity for values > 0. This may produce problems and even bring down some features which ReLU and variants exhibit.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for your comments. But still I am not convinced. log(1+exp(x)), although it is non linear, it does not saturate for x>0. Do you say log(1+exp(x)) has early saturation for x<0 compared to ELU? $\endgroup$
    – user570593
    Jun 10, 2019 at 3:38
  • 1
    $\begingroup$ For all values > 0, log(1+exp(x)) is a non-linear function whereas ReLU is linear ( y = x ). For large negative values, the value of log(1+exp(x)) becomes closer to zero, which indeed is a problem suffered by sigmoid too. $\endgroup$
    – Shubham Panchal
    Jun 10, 2019 at 3:41
  • $\begingroup$ For the potential saturation of negative inputs into ELU, is it less severe than sigmoid (both sides)? Then is parametric ReLU probably better than ELU? $\endgroup$
    – siegfried
    Jan 14, 2022 at 0:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.