# Is using a stop gradient on a residual connection the same as not using the residual connection?

Stop gradient operation prevents the gradients to be calculated for the proceeding graph. However, skip connection outputs are added to the sub-network being skipped over.

• Say $x$ is the identity, and $f(x)$ the main path. A residual block implements $r(x)=x+f(x)$ basically. I think you want to stop the gradient of $f(x)$, right? Aug 6 at 14:40
• According to the question it would be $r(x) = sg(x) + f(x)$, where $sg$ is the stop gradient function. Aug 6 at 22:02

The question is about placing a stop gradient ($$sg$$) operation on the identity path, $$x$$, of a residual block $$r(x)$$, that is: $$r(x) = sg(x) + f(x)$$, where $$f(x)$$ usually represents a couple of convolutions with batch-norm.
In general the stop gradient operation tells auto-diff to treat that expression as a constant, so when asking for the derivative of $$sg(x)$$ it should be the same as differentiating a constant value.
In practice, I think that $$x$$ or, $$sg(x)$$ (since it returns $$x$$ but without tracking its gradients), still accounts for the forward pass of your network but not in the backward pass since backprop would only consider $$f(x)$$ and not also $$x$$.
• No, because with $sg(x)$ (indeed, I assume doing so after each residual block) backpropagation cannot anymore consider the impact of the identity (i.e. its derivative), $x$, which initially is the input (x=input) but afterwards is the output of the $i$-th residual layer, say $x=h_i$. Aug 21 at 13:34