The setting:
We have a neural network $\phi_{\mathbf{w}}:\mathbb{R}^{n} \rightarrow \mathbb{R}^{m}$ with weights $\mathbf{w} \in \mathbb{R}^{q}$. A loss function $\hat{L}: \mathbb{R}^{m} \times \mathbb{R}^{m} \rightarrow \mathbb{R}$ evaluates the quality of a prediction. If $x \in \mathbb{R}^{n}$ shall be mapped to $y \in \mathbb{R}^{m}$ by the neural network, the loss is given as $\hat{L}(\phi(x),y)$.
For a fixed dataset $D \subset \mathbb{R}^{n} \times \mathbb{R}^{m}$, we obtain the empirical error
$F(\mathbf{w}):= \sum_{(x,y) \in D} \hat{L}(\phi_{\mathbf{w}}(x),y)$.
Then $F: \mathbb{R}^{q} \rightarrow \mathbb{R}$.
Now $F$ is minimized using backpropagation.
Let us try to define the vanishing gradient term. I am not sure if there is a proper definition, but I would say we have a vanishing gradient at $p$ if $0 <||\nabla F(p)|| \leq c$ for some small $c$.
Raised questions:
If the gradient is almost zero due to the vanishing gradient, does that mean the current solution is very close to the optimum ? So we can stop iterating..
Why is it bad to have "vanishing gradients" ?
Adressing Question 1
Recall from school that if a functional $F$ has a local optimum at $p$, then $\nabla F(p) = \mathbf{0}$ and $D^2 F(p)$ definite.
If $D^2 F(p)$ is positive definite ($x^T D^2 F(p) x > 0$, for all $x$, where $D^2 F(p)$ is the Hesse matrix), then $p$ is local minimum.
If $\nabla F(p) = \mathbf{0}$ and $D^2 F(p)$ in definite, then $p$ is a saddle point.
In particular this shows that having a zero gradient does not always imply that the position is a local optimum.
(In case of $q = 1$ and $F$ being two times differentiable, $F$ has a local optimum at $p$, if $F'(p) = 0$ and $F''(p) \neq 0 $. )
We can also construct a function that can have arbitrary small gradient while being far away from the minimum:
Consider the function $f_{c}(x) = \max\{0,cx\}$ with $c>0$. Then $\min_{x \in \mathbb{R}} f_c(x) = 0$. For any $p>0$, we have $f'_{c}(p) = c$.
As an example let $p = 10^{9999}$ and $c = 10^{-90}$. Then, the value $f_{c}(p)$ is far away from the minimum, still for the gradient $f'_{c}(p) = 10^{-99}$ holds, which shows that a small gradient does not imply that the current point is close to the optimum.
Adressing Question 2
Note that performing backpropagation is performing the gradient descent algorithm.
Now to address the section questions, there are two directions (an analytical answer and a numerical answer).
The analytical answer would be that a vanishing gradient is nothing special that needs to be considered.
If the step size is choosen appropriately, it can be shown that the sequence of iterates $(p_k)$ is either finite with $\nabla F(p) = 0$, or it is an infinite sequence, and $\lim_{k \rightarrow \infty} \nabla F(p_{k}) = 0$, so that each limit point is a stationary point. This will work independent of any "vanishing gradients".
However, if we consider the question from the numerical aspect, there are certain issues.
1.) There is a machine epsilon $\epsilon$ so that updates with values smaller than $\epsilon$ cannot be performed numerically in a computer. This effectively means that the algorithm converges to some point if $||\nabla F(p)|| \leq \epsilon$.
2.) Even if the values are bigger than $\epsilon$, a "small" gradient vector results in very slow weight updates.
3.) The vanishing gradient problem may arise for example if the sigmoid function is used as activation in a deep neural network. This can be understood from the chain rule.
Let us compute a simple example where we have neural network consisting of $L$ layers, and each layer consists of a single neuron, without any bias. As activation function, we use the sigmoid function $\sigma(t) = \frac{1}{1+e^{-t}}$. Then $\sigma'(t) = \sigma(t)*(1-\sigma(t))$.
The output of the i-th layer is given as
$f_{i}(w) := \sigma(w o_{i-1})$, where $o_{i-1}$ is the output of the $i-1$-th layer.
Let us ignore the loss function and assume that $F$ is exactly the neural network.
The output at layer $i$ is denoted as $o_{i}$ so that we have $o_{i} := \sigma(w_{i} o_{i-1})$ and $o_{1} = w_{1}$.
Then $F(w_{1},\ldots,w_{L}) = o_{L} = \sigma(w_{L} o_{L-1})$.
According to the chain rule, we have $\frac{\mathrm{d}F}{\mathrm{d}w_{1}}(w) = \frac{\mathrm{d} \sigma}{\mathrm{d}w_{1}}(w_{L} o_{L-1}) = \sigma'(w_{L} o_{L-1}) \frac{\mathrm{d} w_{L} o_{L-1}}{\mathrm{d}w_{1}} = \sigma'(w_{L} o_{L-1}) w_{L} \frac{\mathrm{d} o_{L-1}}{\mathrm{d}w_{1}}$.
Repeatedly applying the chain rule results in:
$\frac{\mathrm{d}F}{\mathrm{d}w_{1}}(w) = \prod_{i = 2}^{L} w_{i} \prod_{i = 2}^{L} \sigma'(w_{i}o_{i-1})$.
Since $\sigma(t) \in [0,1]$, we have $\sigma'(t) \in [0,1]$. In particular, we often want $\sigma$ to output either $0$ or $1$. However, if $\sigma(w_{i}o_{i-1})$ is close to $0$ or $1$, then $\sigma'(w_{i}o_{i-1})$ will be close to $0$.
Now if one (or multiply) numbers are close to zero in $\frac{\mathrm{d}F}{\mathrm{d}w_{1}}(w)$, we obtain a very small number, which results in numerical issues (due to reaching machine epsilon), causing a very slow update during the algorithm.
