Do zero weights receive zero gradient in ReLU neural networks?

Question

Suppose I have a deep neural network using the ReLU activation function, that is $\sigma(x) = max(x, 0)$. Suppose some weight $w_i$ becomes exactly $0$ at some point. Am I getting something wrong here, or is it the case that the gradient w.r.t. $w_i$ will be zero at all times and hence $w_i$ won't get any further updates? I feel like I am missing something here.

Answer 1

The derivative (of a loss function) with respect to $w_i$ can be any value, independent of the value of $w_i$. (E.g. $f(x)=x$ has derivative 1 everywhere, even at $x=0$.)