To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated through many layers, the product of these gradients becomes very small.

I know there are other solutions like a rectifier activation function, but my question is why we could not simply use a variation of the often used tanh function.

If the activation function was of the form $\tanh(n x)$ then the maximum possible gradient is $n$. Thus if $n > 1$ we no longer have a case where the product of gradients necessarily goes to 0.

Is there some reason why such an activation function would otherwise fail?

  • $\begingroup$ Well, if you use mini-batches - as common now - in training DNN, then probability/statistics/randomness will enter definitely. One easy-to-understand source for this is Nielsen's book Neural Networks and Deep Learning. How about you run several experiments to verify this yourself and report what you found here after? $\endgroup$ – Tuyen May 11 '19 at 4:47
  • 1
    $\begingroup$ There are quite a few resources that suggest using $1.7159 \text{tanh}(\frac{2}{3}x)$ . . . I have always assumed that was a similar idea to batch normalisation - about creating closer to ideal distribution of features in hidden layers, including the effect on gradients. $\endgroup$ – Neil Slater Oct 7 '19 at 7:50
  • $\begingroup$ @NeilSlater - any references you would suggest looking at? $\endgroup$ – zephyr Oct 7 '19 at 10:49
  • $\begingroup$ Not really, but search for "1.7159 tanh" and you will see it is a commonly used variant $\endgroup$ – Neil Slater Oct 7 '19 at 11:05

You are correct. For $n > 1$, the multiplication of derivatives does not necessarily go to zero, because each derivative could be potentially larger than one (up to $n$).

However, for practical purposes, we should ask ourselves how easy it is to maintain this situation (keeping the multiplication of derivatives away from zero)? Which turns out to be quite hard compared to ReLU, which gives derivative = 1, specially now, when there is also a chance of gradient explosion.


Suppose we have $K$ derivatives (standing for depth $K$) multiplied together as follows $$g=\left.\frac{\partial f(x)}{\partial x} \right|_{x=x_1}\cdots \left.\frac{\partial f(x)}{\partial x} \right|_{x=x_K}$$ each evaluated at different values $x_1$ to $x_K$. In a neural network, each $x_i$ is a weighted sum of outputs $\boldsymbol{h}$ from previous layer, e.g. $x = \boldsymbol{w}^t\boldsymbol{h}$.

As $K$ increases, we want to know what it takes to prevent the vanishing of $g$. For example, for the case of$$f(x)=tanh(x)$$we cannot prevent that because each derivative is smaller than one, except for $x=0$, i.e. $$\frac{\partial f(x)}{\partial x}=\frac{\partial tanh(x)}{\partial x}=1-tanh^2(x) < 1 \text{ for } x \neq 0$$ However, there is a new hope based on your proposal. For $f(x)=tanh(nx)$, derivative could go up to $n > 1$, i.e. $$\frac{\partial f(x)}{\partial x}=\frac{\partial tanh(nx)}{\partial x}=n\left(1-tanh^2(nx)\right) < n \text{ for } x \neq 0.$$

When are forces balanced?

Now, here is the core of my analysis:

How far $x$ needs to move away from $0$ to have a derivative smaller than $\frac{1}{n}$ to cancel out $n$ which is the maximum possible derivative?

The farther $x$ needs to move away from $0$, the harder it is to produce a derivative below $\frac{1}{n}$, thus, the easier it is to prevent the multiplication from vanishing. This question tries to analyze the tension between good $x$'s close to zero, and bad $x$'s far from zero. For example, when good and bad $x$'s are balanced, they would create a situation like $$g=n \times n \times \frac{1}{n} \times n \times \frac{1}{n} \times \frac{1}{n} = 1.$$ For now, I try to be optimistic by not considering arbitrarily large $x_i$'s, since even one of them can bring $g$ arbitrarily close to zero.

For the special case of $n=1$, any $|x| > 0$ results in a derivative $< 1/1 = 1$, therefore, it is almost impossible to keep the balance (prevent $g$ from vanishing) as depth $K$ increases, e.g. $$g=0.99 \times 0.9 \times 0.1 \times 0.995 \cdots \rightarrow 0.$$

For the general case of $n > 1$, we proceed as follows $$\begin{align*} &\frac{\partial tanh(nx)}{\partial x} < \frac{1}{n}\\ &\Rightarrow n\left(1-tanh^2(nx)\right) < \frac{1}{n}\\ &\Rightarrow 1 - \frac{1}{n^2} < tanh^2(nx) \\ &\Rightarrow \sqrt{1 - \frac{1}{n^2}} < |tanh(nx)| \\ &\Rightarrow x > t_1(n):=\frac{1}{n}tanh^{-1}\left(\sqrt{1 - \frac{1}{n^2}}\right) \\ &\text{or } x < t_2(n):=-t_1(n)=\frac{1}{n}tanh^{-1}\left(-\sqrt{1 - \frac{1}{n^2}}\right) \end{align*}$$ So for $|x| > t_1(n)$, the derivative will be smaller than $\frac{1}{n}$. Therefore, in terms of being smaller than one, multiplication of two derivatives at $x_1 \in {\Bbb R}$ and $|x_2| > t_1(n)$ for $n > 1$ is equivalent to an arbitrary derivative for $n=1$, i.e. $$\left(\left.\frac{\partial tanh(nx)}{\partial x}\right|_{x=x_1\in {\Bbb R}} \times \left.\frac{\partial tanh(nx)}{\partial x}\right|_{x = x_2,|x_2| > t_1(n)}\right) \equiv \left.\frac{\partial tanh(x)}{\partial x}\right|_{x = z}, z \in {\Bbb R}\setminus\{0\}.$$ In other words,

$K$ mentioned pairs of derivatives for $n > 1$ is as problematic as $K$ derivatives for $n=1$.

Now, to see how easy (or hard) it is to have $|x| > t_1(n)$, let's plot $t_1(n)$ and $t_2(n)$ (thresholds are plotted for a continuous $n$).

As you can see, to have a derivative $\geq 1/n$, the largest interval is achieved at $n=2$, which is still narrow! This interval is $[-0.658, 0.658]$, meaning for $|x| > 0.658$, the derivative will be smaller than $1/2$. Note: a slightly larger interval is achievable if $n$ is allowed to be continuous.

Based on this analysis, we can now reach a conclusion:

To prevent $g$ from vanishing, around half or more of $x_i$'s need to be inside an interval like $[-0.658, 0.658]$

so when their derivatives are paired with the other half, multiplication of each pair would be above one at best (required that no $x$ is far into large values), i.e. $$\left(\left.\frac{\partial f(x)}{\partial x} \right|_{x=x_1 \in {\Bbb R}}\left. \times \frac{\partial f(x)}{\partial x} \right|_{x=x_2\in [-0.658, 0.658]}\right) > 1$$ However, in practice, it is likely to have more than half of $x$'s outside of $[-0.658, 0.658]$ or a couple of $x$'s with large values, causing $g$ to vanish to zero. Also, there is a problem with too many $x$'s close to zero which is

For $n > 1$, too many $x$'s close to zero could lead to a large gradient $g \gg 1$ (potentially up to $n^K$) which moves (explodes) the weights into larger values ($w_{t+1} = w_{t} + \lambda g$), which further moves the $x$'s into larger values ($x_{t+1} = \boldsymbol{w}^t_{t+1}\boldsymbol{h}_{t+1}$) converting the good $x$'s into (very) bad ones.

How large is too large?

Here, I carry out a similar analysis to see

How far $x$ needs to move away from $0$ to have a derivative smaller than $\frac{1}{n^{K-1}}$ to cancel out the other $K-1$ $x$'s assuming they are very close to zero and acquired the maximum possible gradient?

To answer this question, we derive the below inequality

$$\begin{align*} &\frac{\partial tanh(nx)}{\partial x} < \frac{1}{n^{K-1}} \Rightarrow |x| > \frac{1}{n}tanh^{-1}\left(\sqrt{1 - \frac{1}{n^K}}\right)\\ \end{align*}$$

that shows, for example, for depth $K = 50$ and $n=2$, a value outside $[-9.0, 9.0]$ produces a derivative $< 1/2^{49}$. This result gives an intuition about how easy it is for a couple of $x$'s around 5-10 to cancel out the majority of good $x$'s.

One way street analogy

Based on the previous analyses, I could provide a qualitative analogy using a Markov Chain of two states $[g \gg 0]$ and $[g \sim 0]$ that crudely models the dynamical behavior of gradient $g$ as follows

When system goes to state $[g \sim 0]$, there is not much gradient to bring (change) the values back to state $[g \gg 0]$. This is similar to a one way street that will be passed eventually if we give it enough time (large enough epochs) given convergence of training does not happen (otherwise, we have found a solution before experiencing a vanishing gradient).

A more advanced analysis of dynamical behavior of gradient would be possible by carrying out a simulation on actual neural networks (which potentially depends on many parameters such as loss function, width and depth of network, and data distribution) and come up with

  1. A probabilistic model that tells how often vanishing happens based on a distribution of gradient $g$ or a joint distribution ($\boldsymbol{x}$, $g$) or ($\boldsymbol{w}$, $g$), or
  2. A deterministic model (map) that tells which initial points (initial values of weights) lead to gradient vanishing; possibly accompanied by trajectories from initial to final values.

Exploding gradient problem

We have covered the "vanishing gradient" aspect of $tanh(nx)$. On the contrary, for the "exploding gradient" aspect, we should be worried about having too many $x$'s close to zero, which could potentially produce a gradient around $n^K$, causing numerical instability. For this case, a similar analysis based on inequality $$\begin{align*} &\frac{\partial tanh(nx)}{\partial x} > 1 \Rightarrow |x| < \frac{1}{n}tanh^{-1}\left(\sqrt{1 - \frac{1}{n}}\right)\\ \end{align*}$$ shows that for $n=2$, around half or more of $x_i$'s should be outside of $[-0.441, 0.441]$ to have $g$ around $O(1)$ away from $O(n^K)$. This leaves an even smaller region on ${\Bbb R}^K$ in which $K$ $tanh(nx)$ functions would work well together (neither vanished, nor exploded); reminding that $tanh(x)$ does not have the exploding gradient problem.

| improve this answer | |
  • $\begingroup$ this is a lovely answer - enjoyed reading it! - that said I'm left with one lingering question: as a neural network is trained, and weights and biases vary, is it fair to still see this as a random sample of gradients? - if we limit the starting state to one where the product of gradients is close to unity, is it clear that the network will naturally move away from such a harmonious configuration? $\endgroup$ – zephyr May 10 '19 at 22:36
  • 1
    $\begingroup$ @Zephyr Thanks! I think you are pointing toward a probabilistic or dynamical analysis of the gradient. I do not have a well-articulated answer to these questions, nonetheless I have added some directions in this regard. $\endgroup$ – Esmailian May 11 '19 at 13:20

I have plotted what you are referring in the following picture. As you can see, by employing a coefficient as the input of the tanh function, you are limiting the range of changes of the function with respect to $x$ axis. This has a negative effect. The reason is that although you are making the slope sharper for a very small region in the domain, you are making the differentiation of the other points in the domain more close to zero. The vanishing problem occurs due to the fact that the outputs of neurons go far from the zero and they will be biased to each of the two directions. After that, the differentiation value is so much small and due to begin smaller than one and bigger than zero, it gets even smaller after being multiplied by the other differentiations which are like itself.

enter image description here

Another nice question can be this that you have a coefficient value smaller than one. I've illustrated that In the following picture. In this figure, you are changing the function in a way that you have a differentiation which is larger than before in more points of the domain, but again it is smaller than one. This is not valuable for deep networks.

enter image description here

As I have mentioned, in both cases which you employ coefficient, the derivative would be smaller than one and is not valuable for deep networks.

| improve this answer | |
  • 1
    $\begingroup$ the links don't work $\endgroup$ – oW_ May 7 '19 at 13:50
  • $\begingroup$ i understand your argument - and i think it makes good sense - but it's not clear to me that it's the same problem. With the tanh(x) function the product of many gradients always diminishes, while with (for example) tanh(2x) the product of gradients certainly can dissapear, but they can also take any value (up to 2^n where n is the number of hidden layers) $\endgroup$ – zephyr May 7 '19 at 13:58
  • $\begingroup$ The point is that for $tanh(n)$ it vanishes at $1$ but for $tanh(2n)$ it vanishes at $1/2$. You are limiting the function. After some steps your neurons would be biased to each of the diretions. $\endgroup$ – Media May 7 '19 at 14:00
  • $\begingroup$ this seems an incomplete argument - with a gradient <1 always a single near zero value in the product ensures that the total is close to zero - when the gradient can be greater than 1 a near-zero value can be offset with multiple >1 values - and argument about the tanh(x) case is based on a hard upper limit, but for a tanh(2x) case we fall back on averages, a much less strong logical statement - I'm not saying the above conclusion is incorrect, just that it requires deeper justification - are you aware of any relevant research or examples? $\endgroup$ – zephyr May 8 '19 at 17:01
  • $\begingroup$ I didn'r understand what you want to convey. $\endgroup$ – Media May 8 '19 at 17:39

The derivative of tanh(x) is sech(x)^2, where sech(x)=2e^x/(1+e^(2x)). Hence, when you see the gradient decreases to 0, that means x converges to +/- infinity . If you consider tanh(nx), then the derivative is n sech(nx)^2, and sech(nx)^2 converges to 0 faster than n converging to infinity, when x converges to +/- infinity. Therefore, heuristically, multiplying the argument by bigger n will make things worse.

| improve this answer | |

Thanks to everyone for their great answers - they've really helped in thinking about this problem - and I recommend anyone interested in the problem having a look - but there's a much simpler route to an answer:

When we replace $tanh(x)$ with $tanh(nx)$ as an activation function we have changed nothing about the performance of the activation function.

All we have done is rescaled all the weights and biases of the network - which we are free to do arbitrarily. This will not affect the performance of the network, but certainly will the initialization. Previously I had stated that it will not affect the training either - but I'm now not sure I can state this with full confidence.

| improve this answer | |
  • $\begingroup$ If DNN actually computes the global minimum of the cost function then Yes, probably you are correct. However, in practice, you use gradient descent, and if you use the standard way then you fix a learning rate. Then your claim needs explanation. (Except if you also rescale learning rate and the initial point of the gradient descent process.) $\endgroup$ – Tuyen May 14 '19 at 14:43
  • $\begingroup$ More detail on what I meant: If f(\alpha ) is the cost function provided by the DNN when your activation function is tanh(x), then f(n\alpha ) is the corresponding cost function for the activation function tanh (nx). Here \alpha is the set of parameters of the DNN. Now if you run the (standard) gradient descent method, with learning rate \delta and initial point \alpha _0, then the first two steps are as follows: \alpha _1 = \alpha _0 -\delta n \nabla f(n\alpha _0), and \alpha _2 = \alpha _1-\delta n \nabla f(n\alpha _1). Thus, it is clear that the behaviour of the new cost function ... $\endgroup$ – Tuyen May 15 '19 at 7:41
  • $\begingroup$ ... is not the same as that of the old cost function. Except if you only look at global minima, then it is true that your claim is correct. Otherwise, I don't see how your claim is justified, and more explanation is needed. Even if you rescale the learning rate by n, things are still very much different. $\endgroup$ – Tuyen May 15 '19 at 7:44
  • $\begingroup$ this is an interesting argument and I'm looking into it - will unmark this as the answer for the moment - quick question though - how is this not the same as just rescaling the learning rate, \delta? $\endgroup$ – zephyr May 15 '19 at 8:08
  • 1
    $\begingroup$ I was indeed getting my derivatives wrong - thanks to Tuyen for their patience! $\endgroup$ – zephyr May 15 '19 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.