Stop gradient operation prevents the gradients to be calculated for the proceeding graph. However, skip connection outputs are added to the sub-network being skipped over.
2
-
$\begingroup$ Say $x$ is the identity, and $f(x)$ the main path. A residual block implements $r(x)=x+f(x)$ basically. I think you want to stop the gradient of $f(x)$, right? $\endgroup$– Luca AnzaloneAug 6 at 14:40
-
1$\begingroup$ According to the question it would be $r(x) = sg(x) + f(x)$, where $sg$ is the stop gradient function. $\endgroup$– LightAug 6 at 22:02
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
The question is about placing a stop gradient ($sg$) operation on the identity path, $x$, of a residual block $r(x)$, that is: $r(x) = sg(x) + f(x)$, where $f(x)$ usually represents a couple of convolutions with batch-norm.
In general the stop gradient operation tells auto-diff to treat that expression as a constant, so when asking for the derivative of $sg(x)$ it should be the same as differentiating a constant value.
In practice, I think that $x$ or, $sg(x)$ (since it returns $x$ but without tracking its gradients), still accounts for the forward pass of your network but not in the backward pass since backprop would only consider $f(x)$ and not also $x$.
-
$\begingroup$ So would training be the same as not using a residual connection? $\endgroup$– LightAug 20 at 20:30
-
$\begingroup$ No, because with $sg(x)$ (indeed, I assume doing so after each residual block) backpropagation cannot anymore consider the impact of the identity (i.e. its derivative), $x$, which initially is the input (x=input) but afterwards is the output of the $i$-th residual layer, say $x=h_i$. $\endgroup$ Aug 21 at 13:34