Suppose I have a deep neural network using the ReLU activation function, that is $$\sigma(x) = max(x, 0)$$. Suppose some weight $$w_i$$ becomes exactly $$0$$ at some point. Am I getting something wrong here, or is it the case that the gradient w.r.t. $$w_i$$ will be zero at all times and hence $$w_i$$ won't get any further updates? I feel like I am missing something here.
The derivative (of a loss function) with respect to $$w_i$$ can be any value, independent of the value of $$w_i$$. (E.g. $$f(x)=x$$ has derivative 1 everywhere, even at $$x=0$$.)