Gradients of lower layers of NN when gradient of an upper layer is 0?

Question

Say we have a neural network with an input layer, a hidden layer and an output layer.

Say the gradients with respect to the weights and biases of the output layer are all 0.

Then, by backpropagation and the chain rule, does this necessarily mean that the gradients with respect to the weights and biases of the hidden layer are 0 too?

Answer 1

Summary

1. If we consider a classical feed-forward network with fully connected layers the gradient of a single sample, then your assumption is correct.
2. If we consider the (accumulated) gradients of a batch, then your assumption is not true.
3. If only the gradients of the weights are zero (but we ignore the biases), then your assumption is not true.
4. If you consider other architectures (recurrent networks, convolution, ...) then I would suggest to ask explicitly about them.

Details:

Notion

Let's stick to the following notion:

• $x^{(0)}$ is the input vector
• $W^{(k)}$ is the weight matrix for layer $k=1,2$
• $b^{(k)}$ is the bias vector at layer $k=1,2$
• $z^{(k)}=W^{(k)}x^{(k-1)}+b^{(k)}$ is the raw activation in layer $k$
• $f_k$ is the activation function in layer $k=1,2$
• $x^{(k)} = f_k(z^{(k)})$ is the output of layer $k=1,2$
• $L$ is the loss function, with $L(x^{(2)}, y)$ being the actual loss

Gradients

In the following I assume, that we have a scalar output, i.e. $y, x^{(2)},z^{(2)},b^{(2)}\in\mathbb{R}$ all are scalar values. This makes the notion shorter.

We then get the gradients to the output layers bias $b^{(k)}$:

$$\frac{\partial}{\partial b^{(2)}}L(x^{(2)}, y) = \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y)\cdot \underbrace{\frac{\partial z^{(2)}}{\partial b^{(2)}}}_{=1} = \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y)$$

Hence, if the gradient of the bias of the outut layer is zero (in the singe-sample-case), then the gradient of the raw activation of the output layer is also zero:

$$\frac{\partial}{\partial b}L(x^{(2)}, y) = 0 \quad\Rightarrow\quad \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y) = 0$$

Lets now have a look at the gradient of weight element of the hidden layer: $$\frac{\partial}{\partial W^{(1)}_{i,j}}L(x^{(2)}, y) = \underbrace{\frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y)}_{=0} \cdot \frac{\partial z^{(2)}}{\partial x_i^{(2)}} \cdot \frac{\partial x_i^{(2)}}{\partial W_{i,j}^{(1)}}=0$$

This shows that your assumption is correct, if there is just a single sample and the output bias is 0.

Ignoring the bias

The gradient of the output-layer weights are given by: $$\frac{\partial}{\partial W^{(2)}_{1,j}}L(x^{(2)}, y) = \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y) \cdot \frac{\partial z^{(2)}}{\partial w_{1,j}^{(2)}} = \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y) \cdot x_j^{(2)}$$

So if the hidden layer outputs zeros, the gradient of the weights of the output layer is zero. Nevertheless, the hidden weights might have a non-zero gradient.

Gradients of a batch

The gradient of a batch is computed by accumulation the gradients of the samples (typically by sum or average). This means, that the accumulated gradient of the bias can be zero without the single sample gradients being zero. Do to non-linearity of activation functions, this can mean, that the weights of the hidden layer do not accumulate to zero although the gradients of the output layer do so.