I know that, if X are all positive, or negative then the sign of the downstream gradient will be same as that of the upstream gradient, but what I don't understand is how the zero centered activation function can overcome this problem?
Since, even in the case of tanH function, if all X are positive then too the sign remains same.
Forgive my english, not a native speaker.