Why is vanishing gradient a problem?

Question

Let us say we are using a neural network with $4$ layers with $50,30,20,10$ neurons each. The problem of vanishing gradient would mean that the rate of change of parameters associated with the earlier layer (say first) would be significantly lower than the rate of change of the parameters associated with the later on layers (say 4th).

Now the cost function spans across $n$-dimensions where $n$ is the total number of the parameters and we try to find the minimum of this cost function by varying it across these dimensions.

My question is why does it not lead to the conclusion that the cost function is nearly flat in the dimensions created by the parameters associated with the earlier layers.

Here is the help I can provide on this: this is from neuralnetworksanddeeplearning.com

One response to vanishing (or unstable) gradients is to wonder if they're really such a problem. Momentarily stepping away from neural nets, imagine we were trying to numerically minimize a function $f(x)$ of a single variable. Wouldn't it be good news if the derivative $f′(x)$ was small? Wouldn't that mean we were already near an extremum? In a similar way, might the small gradient in early layers of a deep network mean that we don't need to do much adjustment of the weights and biases?

Of course, this isn't the case. Recall that we randomly initialized the weight and biases in the network. It is extremely unlikely our initial weights and biases will do a good job at whatever it is we want our network to do. To be concrete, consider the first layer of weights in a $[784,30,30,30,10]$ network for the MNIST problem. The random initialization means the first layer throws away most information about the input image. Even if later layers have been extensively trained, they will still find it extremely difficult to identify the input image, simply because they don't have enough information. And so it can't possibly be the case that not much learning needs to be done in the first layer. If we're going to train deep networks, we need to figure out how to address the vanishing gradient problem.

Specific problems from the excerpt:

1. It is extremely unlikely our initial weights and biases will do a good job at whatever it is we want our network to do. Why? Why cant the cost function be nearly flat in the dimensions created by the earlier weights. To be precise why is the explanation not the the last 3 layers are enough to do such a bangup job that they always find weights corresponding to a very small proximity of whatever random weights we initialized the first layer to. Potato pothatoe. i am asking the same thing in different words to make sure i convey myself

2. To be concrete, consider the first layer of weights in a $[784,30,30,30,10]$ network for the MNIST problem. The random initialization means the first layer throws away most information about the input image. What does this mean? Kindly explain. from what i understand it is that we cant trace back the second layer inputs unambiguously to the first layer inputs. but, even if we are initializing the weights randomly, we still know what those weights are so how is it that we are throwing away the information?

Answer 1

why does it not lead to the conclusion that the cost function is nearly flat in the dimensions created by the parameters associated with the earlier layers.

Your conclusion sounds very reasonable - but only in the neighborhood where we calculated the gradient.

E.g. (this is an unrealistic example, but bear with me) consider the case in which the cost function is $C(x,y)=100x^2+y^2$, while $x$ is a weight in a later layer and $y$ is a weight in an earlier layer.
I used https://academo.org/demos/contour-plot/ to draw a contour map of $C$, and then drew $4$ steps (arrows starting at the top left) that a gradient descent algorithm might take.
(For an explanation about contour lines and why they are perpendicular to the gradient, see videos 1 and 2 by the legendary 3Blue1Brown.)

The gradient descent algorithm only sees the slopes where gradients are calculated, so here it thinks that the cost function is quite flat in the dimension of $y$:

Imagine a scenario in which the arrows above are even more densely packed, i.e. the steps make an even smaller progress in the dimension of $y$. This would make the gradient descent algorithm really slow.

Here is another intuitive example for an unfortunate case in which the gradient tells us "the cost function is nearly flat in this dimension, so there is no need to move in that dimension".

$\frac{\partial C}{\partial y}$ is very small here, so a gradient descent algorithm would go right very slowly:

---

As for your other questions, I would try to clarify Nielsen's explanation.

Let's say we have a $[784,30,30,30,10]$ network for the MNIST problem, and it has a severe vanishing gradient problem.
Therefore, after we randomly initialize the weights in the first hidden layer (i.e. the first $30$ neurons layer, whose weight matrix's size is $30\times 784$), they are virtually stuck that way (as the gradient in the earliest layer is extremely small).

Now, consider an unfortunate case in which we randomly initialize the weights in the first hidden layer such that the weights of some pixels are so small, that these pixels' input values are virtually discarded.
To clarify, these input values are multiplied by very small weights, so their weighted value is always close to $0$, regardless of the input value. The weighted value of each input value is the only thing the network knows about the input, and so it is as if these input values were discarded.

In other words, some of the pixels in the input image are invisible to the network.
To demonstrate that, I painted these "invisible pixels" in red:

Let's see how our network sees some MNIST image:

Is it a $6$? Is it a $5$? Even a human like me finds it difficult to say for sure..
What about a picture of an $8$ or $9$? would we be able to identify the digit then?

Nielsen claims that it would be impossible to effectively train such network to identify digits, as it simply has no access to some crucial information - the invisible pixels.