Summary
- If we consider a classical feed-forward network with fully connected layers the gradient of a single sample, then your assumption is correct.
- If we consider the (accumulated) gradients of a batch, then your assumption is not true.
- If only the gradients of the weights are zero (but we ignore the biases), then your assumption is not true.
- If you consider other architectures (recurrent networks, convolution, ...) then I would suggest to ask explicitly about them.
Details:
Notion
Let's stick to the following notion:
- $x^{(0)}$ is the input vector
- $W^{(k)}$ is the weight matrix for layer $k=1,2$
- $b^{(k)}$ is the bias vector at layer $k=1,2$
- $z^{(k)}=W^{(k)}x^{(k-1)}+b^{(k)}$ is the raw activation in layer $k$
- $f_k$ is the activation function in layer $k=1,2$
- $x^{(k)} = f_k(z^{(k)})$ is the output of layer $k=1,2$
- $L$ is the loss function, with $L(x^{(2)}, y)$ being the actual loss
Gradients
In the following I assume, that we have a scalar output, i.e. $y, x^{(2)},z^{(2)},b^{(2)}\in\mathbb{R}$ all are scalar values. This makes the notion shorter.
We then get the gradients to the output layers bias $b^{(k)}$:
$$\frac{\partial}{\partial b^{(2)}}L(x^{(2)}, y) = \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y)\cdot \underbrace{\frac{\partial z^{(2)}}{\partial b^{(2)}}}_{=1} = \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y)$$
Hence, if the gradient of the bias of the outut layer is zero (in the singe-sample-case), then the gradient of the raw activation of the output layer is also zero:
$$\frac{\partial}{\partial b}L(x^{(2)}, y) = 0 \quad\Rightarrow\quad \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y) = 0$$
Lets now have a look at the gradient of weight element of the hidden layer:
$$\frac{\partial}{\partial W^{(1)}_{i,j}}L(x^{(2)}, y) = \underbrace{\frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y)}_{=0} \cdot \frac{\partial z^{(2)}}{\partial x_i^{(2)}} \cdot \frac{\partial x_i^{(2)}}{\partial W_{i,j}^{(1)}}=0$$
This shows that your assumption is correct, if there is just a single sample and the output bias is 0.
Ignoring the bias
The gradient of the output-layer weights are given by:
$$\frac{\partial}{\partial W^{(2)}_{1,j}}L(x^{(2)}, y) = \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y) \cdot \frac{\partial z^{(2)}}{\partial w_{1,j}^{(2)}} = \frac{\partial}{\partial z^{(2)}}L(x^{(2)}, y) \cdot x_j^{(2)}$$
So if the hidden layer outputs zeros, the gradient of the weights of the output layer is zero. Nevertheless, the hidden weights might have a non-zero gradient.
Gradients of a batch
The gradient of a batch is computed by accumulation the gradients of the samples (typically by sum or average). This means, that the accumulated gradient of the bias can be zero without the single sample gradients being zero.
Do to non-linearity of activation functions, this can mean, that the weights of the hidden layer do not accumulate to zero although the gradients of the output layer do so.