4
$\begingroup$

For context, in this paper

Several RNN variants such as the long short-term memory (LSTM) [10, 18] and the gated recurrent unit (GRU) [5] have been proposed to address the gradient problems. However, the use of the hyperbolic tan- gent and the sigmoid functions as the activation function in these variants results in gradient decay over layers. Conse- quently, construction and training of a deep LSTM or GRU based RNN network is practically difficult. By contrast, ex- isting CNNs using non-saturated activation function such as relu can be stacked into a very deep network (e.g. over 20 layers using the basic convolutional layers and over 100 lay- ers with residual connection

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

Please see this answer.

An activation function is considered non-satured if

$$ \lim_{z \rightarrow \infty} f(z) = \infty $$

A saturated activation function has a compact range such as $[-1,1]$ for $\tanh$ or $[0,1]$ for the sigmoid.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ As used in the quote, I agree. However, more generally I think there can be some nuance overlooked for "saturated" vs. "saturating". In most usage, what the quote and this answer refer to should be called "saturating". In comparison, "saturated" implies something about the weights coming into the activation: that the input to the activation function has reached a flat region of the function, so additional changes to the input will have little effect on the output. $\endgroup$
    – Ben Reiniger ♦
    Commented Feb 17, 2019 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.