2
$\begingroup$

I am a beginner in CNN theory and would like to understand the usage of residual modules better. As far as I understand residual modules can be skipped, only the activation function must be computed with the given input.

My question is: How does the network know if it should skip the residual module or not?

$\endgroup$
2
$\begingroup$

Residual blocks contain weights as you can see in this overview of some different variants: enter image description here

Source: this blog post

Since weights are learned parameters the neural net can learn to use or not use non-skip/non-identity paths, i.e. by optimizing with gradient descent the network can learn to skip these blocks (or not).

To phrase it differently: the networks "knows" by following the opposite direction of the gradient.

$\endgroup$
1
  • $\begingroup$ Thank you! This made it clear to me :) $\endgroup$ – Zu Jiry Mar 15 '20 at 9:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.