# Is using a stop gradient on a residual connection the same as not using the residual connection?

Stop gradient operation prevents the gradients to be calculated for the proceeding graph. However, skip connection outputs are added to the sub-network being skipped over.

• Say $x$ is the identity, and $f(x)$ the main path. A residual block implements $r(x)=x+f(x)$ basically. I think you want to stop the gradient of $f(x)$, right? Commented Aug 6, 2023 at 14:40
• According to the question it would be $r(x) = sg(x) + f(x)$, where $sg$ is the stop gradient function. Commented Aug 6, 2023 at 22:02

## 1 Answer

The question is about placing a stop gradient ($$sg$$) operation on the identity path, $$x$$, of a residual block $$r(x)$$, that is: $$r(x) = sg(x) + f(x)$$, where $$f(x)$$ usually represents a couple of convolutions with batch-norm.

In general the stop gradient operation tells auto-diff to treat that expression as a constant, so when asking for the derivative of $$sg(x)$$ it should be the same as differentiating a constant value.

In practice, I think that $$x$$ or, $$sg(x)$$ (since it returns $$x$$ but without tracking its gradients), still accounts for the forward pass of your network but not in the backward pass since backprop would only consider $$f(x)$$ and not also $$x$$.

• So would training be the same as not using a residual connection? Commented Aug 20, 2023 at 20:30
• No, because with $sg(x)$ (indeed, I assume doing so after each residual block) backpropagation cannot anymore consider the impact of the identity (i.e. its derivative), $x$, which initially is the input (x=input) but afterwards is the output of the $i$-th residual layer, say $x=h_i$. Commented Aug 21, 2023 at 13:34