Say we have a neural network with an input layer, a hidden layer and an output layer.

Say the gradients with respect to the weights and biases of the output layer are all 0.

Then, by backpropagation and the chain rule, does this necessarily mean that the gradients with respect to the weights and biases of the hidden layer are 0 too?

  • $\begingroup$ Out of curiosity: why did you ask this or what do you need it for? $\endgroup$
    – Broele
    Commented Jul 23, 2023 at 15:08
  • $\begingroup$ @Broele Thank you for your detailed reply. I want to look into layerwise learning of neural networks, and before that, I thought it would be a good idea to mathematically review backpropagation and exactly how the gradients from one layer affect those in the next. If I create another question, would you be able to provide a description of whether my assumption is true for a CNN? $\endgroup$
    – VJ123
    Commented Jul 23, 2023 at 15:35
  • $\begingroup$ I cannot promise to do so. It always depends on how much time I find: you can imagine that such detailed answers always take some time. But now that I think about it, I would guess that the argument that a bias-gradient of zero leads to a raw-activation-gradient of zero would also hold for CNNs and, hence, also erase all lower layer gradients. $\endgroup$
    – Broele
    Commented Jul 23, 2023 at 15:54
  • $\begingroup$ @Broele Yes, of course, I understand and am already very grateful for your help so far. I shall try and investigate CNNs mathematically - your speculation that they should also work similarly will help me as I attempt to do so. And in case you do find the time, here is a link to the question: datascience.stackexchange.com/questions/122895/…. $\endgroup$
    – VJ123
    Commented Jul 23, 2023 at 16:02

1 Answer 1

  1. If we consider a classical feed-forward network with fully connected layers the gradient of a single sample, then your assumption is correct.
  2. If we consider the (accumulated) gradients of a batch, then your assumption is not true.
  3. If only the gradients of the weights are zero (but we ignore the biases), then your assumption is not true.
  4. If you consider other architectures (recurrent networks, convolution, ...) then I would suggest to ask explicitly about them.

Let's stick to the following notion:

  • $x^{(0)}$ is the input vector
  • $W^{(k)}$ is the weight matrix for layer $k=1,2$
  • $b^{(k)}$ is the bias vector at layer $k=1,2$
  • $z^{(k)}=W^{(k)}x^{(k-1)}+b^{(k)}$ is the raw activation in layer $k$
  • $f_k$ is the activation function in layer $k=1,2$
  • $x^{(k)} = f_k(z^{(k)})$ is the output of layer $k=1,2$
  • $L$ is the loss function, with $L(x^{(2)}, y)$ being the actual loss

In the following I assume, that we have a scalar output, i.e. $y, x^{(2)},z^{(2)},b^{(2)}\in\mathbb{R}$ all are scalar values. This makes the notion shorter.

We then get the gradients to the output layers bias $b^{(k)}$:

$$\frac{\partial}{\partial b^{(2)}}L(x^{(2)}, y) = \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y)\cdot \underbrace{\frac{\partial z^{(2)}}{\partial b^{(2)}}}_{=1} = \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y)$$

Hence, if the gradient of the bias of the outut layer is zero (in the singe-sample-case), then the gradient of the raw activation of the output layer is also zero:

$$\frac{\partial}{\partial b}L(x^{(2)}, y) = 0 \quad\Rightarrow\quad \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y) = 0$$

Lets now have a look at the gradient of weight element of the hidden layer: $$\frac{\partial}{\partial W^{(1)}_{i,j}}L(x^{(2)}, y) = \underbrace{\frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y)}_{=0} \cdot \frac{\partial z^{(2)}}{\partial x_i^{(2)}} \cdot \frac{\partial x_i^{(2)}}{\partial W_{i,j}^{(1)}}=0$$

This shows that your assumption is correct, if there is just a single sample and the output bias is 0.

Ignoring the bias

The gradient of the output-layer weights are given by: $$\frac{\partial}{\partial W^{(2)}_{1,j}}L(x^{(2)}, y) = \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y) \cdot \frac{\partial z^{(2)}}{\partial w_{1,j}^{(2)}} = \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y) \cdot x_j^{(2)}$$

So if the hidden layer outputs zeros, the gradient of the weights of the output layer is zero. Nevertheless, the hidden weights might have a non-zero gradient.

Gradients of a batch

The gradient of a batch is computed by accumulation the gradients of the samples (typically by sum or average). This means, that the accumulated gradient of the bias can be zero without the single sample gradients being zero. Do to non-linearity of activation functions, this can mean, that the weights of the hidden layer do not accumulate to zero although the gradients of the output layer do so.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.