# High level understanding of residual blocks

I've been wondering what the residual blocks actually do and why they are a necessity for deep convolutional networks. Specifically, why are they so useful in some generator networks? To better extract features? I want an intuition for this concept, but can't find an understandable explanation online.

And what does the summation at the end of the block mean? It shows skipping around the block but in the end it adds it separately anyway? What does x and F(x) really represent in this context? Is F(x) an image feature of a given input and x is the input?

To me it looks like it's for feature detection/recognition because of the way it individually adds that separately calculated sub-results, but I could be wrong. And it doesn't entirely make sense to me this way either, because how does this particular method help to achieve that end? Is it a principle of reaffirmation - the net separately calculates some of the features and through these means knows for sure that a certain texture is what it is instead of something else (less mistakes everywhere)? But then - why not have this for the discriminator as well? It fundamentally needs this same tech, doesn't it?

Is it just that the generator needs to improve more than the discriminator does? In other words - if the discriminator had these blocks it would start classifying pretty much only the exact replicas of the input image and the generator would never get fool the discriminator? But then - what importance would that be anyway - the generator would get just as good at image generation anyway.

Suppose we want to fit a function $f(x)$. We can either try to learn a neural network model $F(\cdot)$ so that $F(x) \approx f(x)$. Or, in the residual network approach, we try to learn a neural network model $R(\cdot)$ so that $x+R(x) \approx f(x)$.

Why is the latter easier to learn? There's no fundamental reason why it should necessarily be so, in general. It depends on the specific data set. However, one possible intuition is that we might expect that in some settings, $f(\cdot)$ might be approximately linear: i.e., as a first-order approximation, $f(x) \approx x$ might be a reasonable first-order approximation. Then we want to model the error term: i.e., suppose $f(x)=x+r(x)$; then we want to model $r(x)$. In some settings, the error term $r(x)$ might be simpler to model or smaller than $f(x)$. In that case, a residual network architecture might work better.

If you like, you can think of this as being akin to a Taylor-series approximation. The Taylor series for $f(x)$ is

$$f(x) = c_0 + c_1 x + c_2 x^2 + \cdots$$

Suppose $x$ is small (i.e., $|x| \ll 1$). Then $f(x) = c_0$ is a zero-th order approximation; $f(x) = c_0 + c_1 x$ is a first-order approximation; and so on. At each step, we expect that the residual/error term is probably smaller than the approximation. You can think of a residual network as learning a Taylor series for $f(x)$, in the special case where $c_0 = 0$ and $c_1 = 1$.

There are other more sophisticated explanations / intuitions, but hopefully this gives one possible perspective.

• Is the linearity of f(x) dependant on the activation function, i.e. would a ReLU based network be linear? I didn't understand the Taylor series analogy. Is c1*x a residual block and c2*x the next one? (Which would mean they decrease in "importance" over time) Also, isn't the "error term" always smaller than f(x)? May 22, 2017 at 7:37
• @Thorra, no, I don't think this is related to the choice of activation function (though ReLU seems to work better for optimization, so you probably want to use every trick you can to make things work better -- if you're bothering to use advanced concepts like ResNet you probably want to use ReLU, too). The Taylor series analogy is more of a metaphor, about successive approximation, rather than something to take too literally.
– D.W.
May 22, 2017 at 15:25