Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I know that, if X are all positive, or negative then the sign of the downstream gradient will be same as that of the upstream gradient, but what I don't understand is how the zero centered activation function can overcome this problem?
Since, even in the case of tanH function, if all X are positive then too the sign remains same.
Forgive my english, not a native speaker.
$\tanh$ function is scaled standard sigmoid function $y=\frac{1}{1+e^{-ax}}$. Due to that scaling it has more steep gradient that standard sigmoid function. Steep gradient is important, because it makes backprop training faster and less likely to get stuck in zero-gradient area.
Furthermore $y(0) \neq 0$ where $\tanh(0) = 0$. Sometimes it is very important, especially when dealing with normalized $[0,1]$ signals as inputs.
You can have a look at this survey: https://arxiv.org/pdf/2004.06632.pdf It discusses different aspects of activation functions. It also explains why centered activation functions are considered to be more suitable in practice.
Note that if you consider the universal approximation theorem, the activation function does not need to be zero-centered.
