0
$\begingroup$

Stop gradient operation prevents the gradients to be calculated for the proceeding graph. However, skip connection outputs are added to the sub-network being skipped over.

$\endgroup$
2
  • $\begingroup$ Say $x$ is the identity, and $f(x)$ the main path. A residual block implements $r(x)=x+f(x)$ basically. I think you want to stop the gradient of $f(x)$, right? $\endgroup$ Commented Aug 6, 2023 at 14:40
  • 1
    $\begingroup$ According to the question it would be $r(x) = sg(x) + f(x)$, where $sg$ is the stop gradient function. $\endgroup$
    – Light
    Commented Aug 6, 2023 at 22:02

1 Answer 1

0
$\begingroup$

The question is about placing a stop gradient ($sg$) operation on the identity path, $x$, of a residual block $r(x)$, that is: $r(x) = sg(x) + f(x)$, where $f(x)$ usually represents a couple of convolutions with batch-norm.

In general the stop gradient operation tells auto-diff to treat that expression as a constant, so when asking for the derivative of $sg(x)$ it should be the same as differentiating a constant value.

In practice, I think that $x$ or, $sg(x)$ (since it returns $x$ but without tracking its gradients), still accounts for the forward pass of your network but not in the backward pass since backprop would only consider $f(x)$ and not also $x$.

$\endgroup$
2
  • $\begingroup$ So would training be the same as not using a residual connection? $\endgroup$
    – Light
    Commented Aug 20, 2023 at 20:30
  • $\begingroup$ No, because with $sg(x)$ (indeed, I assume doing so after each residual block) backpropagation cannot anymore consider the impact of the identity (i.e. its derivative), $x$, which initially is the input (x=input) but afterwards is the output of the $i$-th residual layer, say $x=h_i$. $\endgroup$ Commented Aug 21, 2023 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.