# Why does degradation occur in deep neural networks?

It has been shown that "plain" neural networks tend to have an increased amount training error, and accompanied test error, as more layers are added. I am not quite certain as to why this occurs. In the original ResNet paper they hypothesize and verify that this is not due to vanishing gradient.

From what I understand, it is difficult for a model to approximate the identity map between layers and furthermore when this map is optimal the model may tend to approximate the zero function instead. If this is the case, why does this occur? Finally, why does this not occur in shallower networks in which the identity map may also be optimal?

I am new to this topic so I apologize if I may not fully understand this problem. Thank you.

The degradation problem has been observed while training deep neural networks. As we increase network depth, accuracy gets saturated (this is expected). Why is this expected? Because we expect a sufficiently deep neural network to model all the intricacies of our data well. There will come a time, we thought, when the extra modelling power provided to us by the additional layers in a deep network will completely learn our data.

This was the easy part. Now we also saw that as we increased the layers of our network further (after the saturation region), the accuracy of the network dropped. Okay, we say, this could be due to overfitting. Except, it’s not due to overfitting, and additional layers in a deep model lead to higher training errors (training, not testing)!

As you can see in the graph above, deeper networks lead to higher training error. To appreciate how counterintuitive this finding is, consider the following argument.

Consider a network having n layers. This network produces some training error. Now consider a deeper network with m layers (m>n) . When we train this network, we expect it to perform at least as well as the shallower network. Why? Replace the first n layers of the deep network with the trained n layers of the shallower network. Now replace the remaining n−m layers in the deeper network with an identity mapping (that is, these layers simply output what is fed into them without changing it in anyway). Thus, our deeper model can easily learn the shallower model’s representation. If there exists a more complex representation of data, we expect the deep model to learn this. See note at the end.

But this doesn’t happen in practice! We just saw above that deeper networks lead to higher training error!

This is the problem residual networks aim to solve.

Note: An analogy to understand this is from polynomial regression. Let’s say I have some data which can be learned effectively using a linear representation, that is, my hypothesis is h(x)=wx+b where w and b are learned parameters. To be extra sure, I use the quadratic hypothesis h(x)=ax2+bx+c while training. Now if the linear hypothesis is the best way to learn this data, I expect the quadratic hypothesis to learn this linear representation by learning that a−>0 . This is what is observed in practise, but as we saw above, doesn’t apply to neural networks!

## Why do deep networks need to learn identity functions?

The authors talk about linear functions because they are thinking "If a network with fewer layers should work, why should a network with more layers perform worse? After all, if we add more layers that just compute the identity function, the shallower network should have the same output as the longer network. So clearly deeper networks shouldn't be WORSE than shallower networks."

So you are absolutely right: "It is difficult for a model to approximate the identity map between layers." And because this is difficult, deeper networks cannot just learn identity functions to match the performance of shallower networks.

## Why is it difficult to learn the identity function in a layer?

The identity function is hard for a typical deep network layer to learn because the typical layer is non-linear. But the identity function is a linear function. All those ReLUs and Softmaxes that put kinks into the function by design make it hard to learn a function without kinks.

This is where your guesses start to get off track. "When this map is optimal the model may tend to approximate the zero function instead" Actually, the model is likely to learn some wierd approximation of a straight line with a kink in it somewhere.

So why the talk about learning the zero function? A residual layer learns to output $$g(x) = x + f(x)$$, where $$f(x)$$ is the weird nonlinear function with kinks. But if $$f(x)$$ learns the zero function $$f(x) = 0$$, then the whole layer $$g(x)$$ learns the identity function $$g(x) = x$$.

It is a lot easier for a nonlinear function to learn the zero function than the identity function.

# Why don't shallower networks need the identity function?

Because they are shallow enough that it isn't required. There are plenty of nonlinearities for them to learn that will continue to fit the nonlinear function they are learning well.

Follow-up questions welcome.