ReLU and all its variants ( except ReLU-6 ) are linear i.e $ y = x $ for values greater than or equal to 0.
This gives ReLU and specifically ELU an advantage like:
- Linearity means that the slope does not plateau or saturate when $x$ becomes larger. Hence, the vanishing gradient problem is solved.
Now, the graph $ y = log( 1 + e^x ) $ isn't linear for values > 0.
For larger negative values, the graph produces values which are very close to zero. This is also found in sigmoid where larger values produce a fully saturated activation. Hence, $ y = log( 1 + e^x ) $ can raise problems which sigmoid and tanh suffer.
ELU has a log curve for all negative values which is $ y = \alpha( e^x - 1 )$. It does not produce a saturated firing for some extent but saturates for larger negative values.
See here for more information.
Hence, $ y = log( 1 + e^x ) $ is not used because of early saturation
for negative values and also non linearity for values > 0. This may
produce problems and even bring down some features which ReLU and