You are correct. For $n > 1$, the multiplication of derivatives does not necessarily go to zero, because each derivative could be potentially larger than one (up to $n$).
However, for practical purposes, we should ask ourselves how easy it is to maintain this situation (keeping the multiplication of derivatives away from zero)? Which turns out to be quite hard compared to ReLU, which gives derivative = 1, specially now, when there is also a chance of gradient explosion.
Introduction
Suppose we have $K$ derivatives (standing for depth $K$) multiplied together as follows
$$g=\left.\frac{\partial f(x)}{\partial x} \right|_{x=x_1}\cdots \left.\frac{\partial f(x)}{\partial x} \right|_{x=x_K}$$
each evaluated at different values $x_1$ to $x_K$. In a neural network, each $x_i$ is a weighted sum of outputs $\boldsymbol{h}$ from previous layer, e.g. $x = \boldsymbol{w}^t\boldsymbol{h}$.
As $K$ increases, we want to know what it takes to prevent the vanishing of $g$. For example, for the case of$$f(x)=tanh(x)$$we cannot prevent that because each derivative is smaller than one, except for $x=0$, i.e.
$$\frac{\partial f(x)}{\partial x}=\frac{\partial tanh(x)}{\partial x}=1-tanh^2(x) < 1 \text{ for } x \neq 0$$
However, there is a new hope based on your proposal. For $f(x)=tanh(nx)$, derivative could go up to $n > 1$, i.e.
$$\frac{\partial f(x)}{\partial x}=\frac{\partial tanh(nx)}{\partial x}=n\left(1-tanh^2(nx)\right) < n \text{ for } x \neq 0.$$
When are forces balanced?
Now, here is the core of my analysis:
How far $x$ needs to move away from $0$ to have a derivative smaller than
$\frac{1}{n}$ to cancel out $n$ which is the maximum possible derivative?
The farther $x$ needs to move away from $0$, the harder it is to produce a derivative below $\frac{1}{n}$, thus, the easier it is to prevent the multiplication from vanishing. This question tries to analyze the tension between good $x$'s close to zero, and bad $x$'s far from zero. For example, when good and bad $x$'s are balanced, they would create a situation like
$$g=n \times n \times \frac{1}{n} \times n \times \frac{1}{n} \times \frac{1}{n} = 1.$$
For now, I try to be optimistic by not considering arbitrarily large $x_i$'s, since even one of them can bring $g$ arbitrarily close to zero.
For the special case of $n=1$, any $|x| > 0$ results in a derivative $< 1/1 = 1$, therefore, it is almost impossible to keep the balance (prevent $g$ from vanishing) as depth $K$ increases, e.g.
$$g=0.99 \times 0.9 \times 0.1 \times 0.995 \cdots \rightarrow 0.$$
For the general case of $n > 1$, we proceed as follows
$$\begin{align*}
&\frac{\partial tanh(nx)}{\partial x} < \frac{1}{n}\\
&\Rightarrow n\left(1-tanh^2(nx)\right) < \frac{1}{n}\\
&\Rightarrow 1 - \frac{1}{n^2} < tanh^2(nx) \\
&\Rightarrow \sqrt{1 - \frac{1}{n^2}} < |tanh(nx)| \\
&\Rightarrow x > t_1(n):=\frac{1}{n}tanh^{-1}\left(\sqrt{1 - \frac{1}{n^2}}\right) \\
&\text{or } x < t_2(n):=-t_1(n)=\frac{1}{n}tanh^{-1}\left(-\sqrt{1 - \frac{1}{n^2}}\right)
\end{align*}$$
So for $|x| > t_1(n)$, the derivative will be smaller than $\frac{1}{n}$. Therefore, in terms of being smaller than one, multiplication of two derivatives at $x_1 \in {\Bbb R}$ and $|x_2| > t_1(n)$ for $n > 1$ is equivalent to an arbitrary derivative for $n=1$, i.e.
$$\left(\left.\frac{\partial tanh(nx)}{\partial x}\right|_{x=x_1\in {\Bbb R}} \times \left.\frac{\partial tanh(nx)}{\partial x}\right|_{x = x_2,|x_2| > t_1(n)}\right) \equiv \left.\frac{\partial tanh(x)}{\partial x}\right|_{x = z}, z \in {\Bbb R}\setminus\{0\}.$$
In other words,
$K$ mentioned pairs of derivatives for $n > 1$ is as problematic as $K$
derivatives for $n=1$.
Now, to see how easy (or hard) it is to have $|x| > t_1(n)$, let's plot $t_1(n)$ and $t_2(n)$ (thresholds are plotted for a continuous $n$).
As you can see, to have a derivative $\geq 1/n$, the largest interval is achieved at $n=2$, which is still narrow! This interval is $[-0.658, 0.658]$, meaning for $|x| > 0.658$, the derivative will be smaller than $1/2$. Note: a slightly larger interval is achievable if $n$ is allowed to be continuous.
Based on this analysis, we can now reach a conclusion:
To prevent $g$ from vanishing, around half or more of $x_i$'s need to be
inside an interval like $[-0.658, 0.658]$
so when their derivatives are paired with the other half, multiplication of each pair would be above one at best (required that no $x$ is far into large values), i.e.
$$\left(\left.\frac{\partial f(x)}{\partial x} \right|_{x=x_1 \in {\Bbb R}}\left. \times \frac{\partial f(x)}{\partial x} \right|_{x=x_2\in [-0.658, 0.658]}\right) > 1$$
However, in practice, it is likely to have more than half of $x$'s outside of $[-0.658, 0.658]$ or a couple of $x$'s with large values, causing $g$ to vanish to zero. Also, there is a problem with too many $x$'s close to zero which is
For $n > 1$, too many $x$'s close to zero could lead to a large gradient $g \gg 1$ (potentially up to $n^K$) which moves (explodes) the
weights into larger values ($w_{t+1} = w_{t} + \lambda g$), which
further moves the $x$'s into larger values ($x_{t+1} = \boldsymbol{w}^t_{t+1}\boldsymbol{h}_{t+1}$) converting the good $x$'s into (very) bad ones.
How large is too large?
Here, I carry out a similar analysis to see
How far $x$ needs to move away from $0$ to have a derivative smaller than
$\frac{1}{n^{K-1}}$ to cancel out the other $K-1$ $x$'s assuming they are very close to zero and acquired the maximum possible gradient?
To answer this question, we derive the below inequality
$$\begin{align*}
&\frac{\partial tanh(nx)}{\partial x} < \frac{1}{n^{K-1}} \Rightarrow |x| > \frac{1}{n}tanh^{-1}\left(\sqrt{1 - \frac{1}{n^K}}\right)\\
\end{align*}$$
that shows, for example, for depth $K = 50$ and $n=2$, a value outside $[-9.0, 9.0]$ produces a derivative $< 1/2^{49}$. This result gives an intuition about how easy it is for a couple of $x$'s around 5-10 to cancel out the majority of good $x$'s.
One way street analogy
Based on the previous analyses, I could provide a qualitative analogy using a Markov Chain of two states $[g \gg 0]$ and $[g \sim 0]$ that crudely models the dynamical behavior of gradient $g$ as follows
When system goes to state $[g \sim 0]$, there is not much gradient to bring (change) the values back to state $[g \gg 0]$. This is similar to a one way street that will be passed eventually if we give it enough time (large enough epochs) given convergence of training does not happen (otherwise, we have found a solution before experiencing a vanishing gradient).
A more advanced analysis of dynamical behavior of gradient would be possible by carrying out a simulation on actual neural networks (which potentially depends on many parameters such as loss function, width and depth of network, and data distribution) and come up with
- A probabilistic model that tells how often vanishing happens based on a distribution of gradient $g$ or a joint distribution ($\boldsymbol{x}$, $g$) or ($\boldsymbol{w}$, $g$), or
- A deterministic model (map) that tells which initial points (initial values of weights) lead to gradient vanishing; possibly accompanied by trajectories from initial to final values.
Exploding gradient problem
We have covered the "vanishing gradient" aspect of $tanh(nx)$. On the contrary, for the "exploding gradient" aspect, we should be worried about having too many $x$'s close to zero, which could potentially produce a gradient around $n^K$, causing numerical instability. For this case, a similar analysis based on inequality
$$\begin{align*}
&\frac{\partial tanh(nx)}{\partial x} > 1 \Rightarrow |x| < \frac{1}{n}tanh^{-1}\left(\sqrt{1 - \frac{1}{n}}\right)\\
\end{align*}$$
shows that for $n=2$, around half or more of $x_i$'s should be outside of $[-0.441, 0.441]$ to have $g$ around $O(1)$ away from $O(n^K)$. This leaves an even smaller region on ${\Bbb R}^K$ in which $K$ $tanh(nx)$ functions would work well together (neither vanished, nor exploded); reminding that $tanh(x)$ does not have the exploding gradient problem.
