Let us say we are using a neural network with $4$ layers with $50,30,20,10$ neurons each. The problem of vanishing gradient would mean that the rate of change of parameters associated with the earlier layer (say first) would be significantly lower than the rate of change of the parameters associated with the later on layers (say 4th).
Now the cost function spans across $n$-dimensions where $n$ is the total number of the parameters and we try to find the minimum of this cost function by varying it across these dimensions.
My question is why does it not lead to the conclusion that the cost function is nearly flat in the dimensions created by the parameters associated with the earlier layers.
Here is the help I can provide on this: this is from neuralnetworksanddeeplearning.com
One response to vanishing (or unstable) gradients is to wonder if they're really such a problem. Momentarily stepping away from neural nets, imagine we were trying to numerically minimize a function $f(x)$ of a single variable. Wouldn't it be good news if the derivative $f′(x)$ was small? Wouldn't that mean we were already near an extremum? In a similar way, might the small gradient in early layers of a deep network mean that we don't need to do much adjustment of the weights and biases?
Of course, this isn't the case. Recall that we randomly initialized the weight and biases in the network. It is extremely unlikely our initial weights and biases will do a good job at whatever it is we want our network to do. To be concrete, consider the first layer of weights in a $[784,30,30,30,10]$ network for the MNIST problem. The random initialization means the first layer throws away most information about the input image. Even if later layers have been extensively trained, they will still find it extremely difficult to identify the input image, simply because they don't have enough information. And so it can't possibly be the case that not much learning needs to be done in the first layer. If we're going to train deep networks, we need to figure out how to address the vanishing gradient problem.
Specific problems from the excerpt:
It is extremely unlikely our initial weights and biases will do a good job at whatever it is we want our network to do. Why? Why cant the cost function be nearly flat in the dimensions created by the earlier weights. To be precise why is the explanation not the the last 3 layers are enough to do such a bangup job that they always find weights corresponding to a very small proximity of whatever random weights we initialized the first layer to. Potato pothatoe. i am asking the same thing in different words to make sure i convey myself
To be concrete, consider the first layer of weights in a $[784,30,30,30,10]$ network for the MNIST problem. The random initialization means the first layer throws away most information about the input image. What does this mean? Kindly explain. from what i understand it is that we cant trace back the second layer inputs unambiguously to the first layer inputs. but, even if we are initializing the weights randomly, we still know what those weights are so how is it that we are throwing away the information?