2
$\begingroup$

I am a beginner in CNN theory and would like to understand the usage of residual modules better. As far as I understand residual modules can be skipped, only the activation function must be computed with the given input.

My question is: How does the network know if it should skip the residual module or not?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Residual blocks contain weights as you can see in this overview of some different variants: enter image description here

Source: this blog post

Since weights are learned parameters the neural net can learn to use or not use non-skip/non-identity paths, i.e. by optimizing with gradient descent the network can learn to skip these blocks (or not).

To phrase it differently: the networks "knows" by following the opposite direction of the gradient.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you! This made it clear to me :) $\endgroup$
    – Zu Jiry
    Commented Mar 15, 2020 at 9:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.