Neural Network stacking layers

Question

I was watching Andrew Ng's video on ResNets, and he mentioned that "And in theory, as you make a neural network deeper, it should only do better and better on the training set." Here is my understanding of neural networks, as the model progresses through the model, the parameters it will learn will become more and more sophisticated, correct? Intuitively, it should be able to recognize/discover more detailed pattern information about the training set. Is my understanding correct?

Then why, in practice, does adding excessive layers to Neural Networks actually harm the performance of the model? Thanks in advance.

Answer 1

The answer given so far is wrong.

1.) In general, adding more and more layers does not lead to better training accuracy.

2.) The issue is certainly not overfitting, as Andrew Ng is talking only about the training set.

The answer becomes true if you add layers, where each additional layer $l$ is represented by a mapping $f^{(l)}_{w}: \mathbb{R}^{n_{l}} \rightarrow \mathbb{R}^{n_{l}}$ for which weights $w$ exist, such that $f^{(l)}_{w}$ is the identity map. In such a case, you can add arbitrary many layers to an existing neural network architecture and in theory optimizing this new network cannot be worse than without the additional layers (since all additional layers can become the identity map). Potentially these new layers could improve the training accuracy.

In practice this is not observed (see the ResNet paper). The reasons are mainly

1.) The employed solvers do not deliver the global optimal solution.

2.) Adding more layers makes the gradient computation more prone to numerical errors due to the involved chain rule, e.g. have a look here.

Answer 2

Adding more and more layer indeed make the network learn better and better on the training set. However, this causes a problem called "overfit". Overfitting means that your model works extremely well on training set but works poorly on validation set or testing set.

For your last question, the performance of Neural Network models is measured by the ability of the model to predict correctly for unseen data (or future data), which is the accuracy of predicting testing data. In practice, when you make your model deeper, the model will fit more completely to the training data. In this case, you increase the chance of getting an overfitted model. Therefore, the performance of the model (which is the accuracy of predicting testing data) will decrease due to overfitting into training data.

Reference: Overfitting