There is a well known problem vanishing gradient in BackPropagation training of Feedforward Neural Network (FNN)(here we don't consider the vanishing gradient of Recurrent Neural Network).

I don't understand why vanishing gradient does not mean the zero gradient namely the optimal solution we want? I saw some answer said vanishing gradient is not exactly the zero gradient, just means the update of gradient is very slow. However, the stopping rule of gradient decent is just the unchange of parameter within $\epsilon.$

So can anyone give me a clear answer?


1 Answer 1


The setting:

We have a neural network $\phi_{\mathbf{w}}:\mathbb{R}^{n} \rightarrow \mathbb{R}^{m}$ with weights $\mathbf{w} \in \mathbb{R}^{q}$. A loss function $\hat{L}: \mathbb{R}^{m} \times \mathbb{R}^{m} \rightarrow \mathbb{R}$ evaluates the quality of a prediction. If $x \in \mathbb{R}^{n}$ shall be mapped to $y \in \mathbb{R}^{m}$ by the neural network, the loss is given as $\hat{L}(\phi(x),y)$.

For a fixed dataset $D \subset \mathbb{R}^{n} \times \mathbb{R}^{m}$, we obtain the empirical error

$F(\mathbf{w}):= \sum_{(x,y) \in D} \hat{L}(\phi_{\mathbf{w}}(x),y)$. Then $F: \mathbb{R}^{q} \rightarrow \mathbb{R}$.

Now $F$ is minimized using backpropagation.

Let us try to define the vanishing gradient term. I am not sure if there is a proper definition, but I would say we have a vanishing gradient at $p$ if $0 <||\nabla F(p)|| \leq c$ for some small $c$.

Raised questions:

  • If the gradient is almost zero due to the vanishing gradient, does that mean the current solution is very close to the optimum ? So we can stop iterating..

  • Why is it bad to have "vanishing gradients" ?

Adressing Question 1

Recall from school that if a functional $F$ has a local optimum at $p$, then $\nabla F(p) = \mathbf{0}$ and $D^2 F(p)$ definite.

If $D^2 F(p)$ is positive definite ($x^T D^2 F(p) x > 0$, for all $x$, where $D^2 F(p)$ is the Hesse matrix), then $p$ is local minimum.

If $\nabla F(p) = \mathbf{0}$ and $D^2 F(p)$ in definite, then $p$ is a saddle point.

In particular this shows that having a zero gradient does not always imply that the position is a local optimum.

(In case of $q = 1$ and $F$ being two times differentiable, $F$ has a local optimum at $p$, if $F'(p) = 0$ and $F''(p) \neq 0 $. )

We can also construct a function that can have arbitrary small gradient while being far away from the minimum: Consider the function $f_{c}(x) = \max\{0,cx\}$ with $c>0$. Then $\min_{x \in \mathbb{R}} f_c(x) = 0$. For any $p>0$, we have $f'_{c}(p) = c$.

As an example let $p = 10^{9999}$ and $c = 10^{-90}$. Then, the value $f_{c}(p)$ is far away from the minimum, still for the gradient $f'_{c}(p) = 10^{-99}$ holds, which shows that a small gradient does not imply that the current point is close to the optimum.

Adressing Question 2

Note that performing backpropagation is performing the gradient descent algorithm.

Now to address the section questions, there are two directions (an analytical answer and a numerical answer).

The analytical answer would be that a vanishing gradient is nothing special that needs to be considered.

If the step size is choosen appropriately, it can be shown that the sequence of iterates $(p_k)$ is either finite with $\nabla F(p) = 0$, or it is an infinite sequence, and $\lim_{k \rightarrow \infty} \nabla F(p_{k}) = 0$, so that each limit point is a stationary point. This will work independent of any "vanishing gradients".

However, if we consider the question from the numerical aspect, there are certain issues.

1.) There is a machine epsilon $\epsilon$ so that updates with values smaller than $\epsilon$ cannot be performed numerically in a computer. This effectively means that the algorithm converges to some point if $||\nabla F(p)|| \leq \epsilon$.

2.) Even if the values are bigger than $\epsilon$, a "small" gradient vector results in very slow weight updates.

3.) The vanishing gradient problem may arise for example if the sigmoid function is used as activation in a deep neural network. This can be understood from the chain rule. Let us compute a simple example where we have neural network consisting of $L$ layers, and each layer consists of a single neuron, without any bias. As activation function, we use the sigmoid function $\sigma(t) = \frac{1}{1+e^{-t}}$. Then $\sigma'(t) = \sigma(t)*(1-\sigma(t))$.

The output of the i-th layer is given as $f_{i}(w) := \sigma(w o_{i-1})$, where $o_{i-1}$ is the output of the $i-1$-th layer.

Let us ignore the loss function and assume that $F$ is exactly the neural network.

The output at layer $i$ is denoted as $o_{i}$ so that we have $o_{i} := \sigma(w_{i} o_{i-1})$ and $o_{1} = w_{1}$.

Then $F(w_{1},\ldots,w_{L}) = o_{L} = \sigma(w_{L} o_{L-1})$.

According to the chain rule, we have $\frac{\mathrm{d}F}{\mathrm{d}w_{1}}(w) = \frac{\mathrm{d} \sigma}{\mathrm{d}w_{1}}(w_{L} o_{L-1}) = \sigma'(w_{L} o_{L-1}) \frac{\mathrm{d} w_{L} o_{L-1}}{\mathrm{d}w_{1}} = \sigma'(w_{L} o_{L-1}) w_{L} \frac{\mathrm{d} o_{L-1}}{\mathrm{d}w_{1}}$.

Repeatedly applying the chain rule results in:

$\frac{\mathrm{d}F}{\mathrm{d}w_{1}}(w) = \prod_{i = 2}^{L} w_{i} \prod_{i = 2}^{L} \sigma'(w_{i}o_{i-1})$.

Since $\sigma(t) \in [0,1]$, we have $\sigma'(t) \in [0,1]$. In particular, we often want $\sigma$ to output either $0$ or $1$. However, if $\sigma(w_{i}o_{i-1})$ is close to $0$ or $1$, then $\sigma'(w_{i}o_{i-1})$ will be close to $0$.

Now if one (or multiply) numbers are close to zero in $\frac{\mathrm{d}F}{\mathrm{d}w_{1}}(w)$, we obtain a very small number, which results in numerical issues (due to reaching machine epsilon), causing a very slow update during the algorithm.

  • $\begingroup$ Thanks. by the way, what's the difference between 'Data Science' and 'Cross Validated'? Which one should be better to ask the questions of machine learning? $\endgroup$ Oct 2, 2020 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.