# What does it mean for an activation function to be “saturated/non-saturated”?

For context, in this paper

Several RNN variants such as the long short-term memory (LSTM) [10, 18] and the gated recurrent unit (GRU) [5] have been proposed to address the gradient problems. However, the use of the hyperbolic tan- gent and the sigmoid functions as the activation function in these variants results in gradient decay over layers. Conse- quently, construction and training of a deep LSTM or GRU based RNN network is practically difficult. By contrast, ex- isting CNNs using non-saturated activation function such as relu can be stacked into a very deep network (e.g. over 20 layers using the basic convolutional layers and over 100 lay- ers with residual connection

$$\lim_{z \rightarrow \infty} f(z) = \infty$$
A saturated activation function has a compact range such as $$[-1,1]$$ for $$\tanh$$ or $$[0,1]$$ for the sigmoid.