I was watching Andrew Ng's video on ResNets, and he mentioned that "And in theory, as you make a neural network deeper, it should only do better and better on the training set." Here is my understanding of neural networks, as the model progresses through the model, the parameters it will learn will become more and more sophisticated, correct? Intuitively, it should be able to recognize/discover more detailed pattern information about the training set. Is my understanding correct?

Then why, in practice, does adding excessive layers to Neural Networks actually harm the performance of the model? Thanks in advance.


The answer given so far is wrong.

1.) In general, adding more and more layers does not lead to better training accuracy.

2.) The issue is certainly not overfitting, as Andrew Ng is talking only about the training set.

The answer becomes true if you add layers, where each additional layer $l$ is represented by a mapping $f^{(l)}_{w}: \mathbb{R}^{n_{l}} \rightarrow \mathbb{R}^{n_{l}}$ for which weights $w$ exist, such that $f^{(l)}_{w}$ is the identity map. In such a case, you can add arbitrary many layers to an existing neural network architecture and in theory optimizing this new network cannot be worse than without the additional layers (since all additional layers can become the identity map). Potentially these new layers could improve the training accuracy.

In practice this is not observed (see the ResNet paper). The reasons are mainly

1.) The employed solvers do not deliver the global optimal solution.

2.) Adding more layers makes the gradient computation more prone to numerical errors due to the involved chain rule, e.g. have a look here.


Adding more and more layer indeed make the network learn better and better on the training set. However, this causes a problem called "overfit". Overfitting means that your model works extremely well on training set but works poorly on validation set or testing set.

For your last question, the performance of Neural Network models is measured by the ability of the model to predict correctly for unseen data (or future data), which is the accuracy of predicting testing data. In practice, when you make your model deeper, the model will fit more completely to the training data. In this case, you increase the chance of getting an overfitted model. Therefore, the performance of the model (which is the accuracy of predicting testing data) will decrease due to overfitting into training data.

Reference: Overfitting

  • $\begingroup$ Just to make sure, the model 'learns better and better' on the training set by casting layers of non-linearity activation functions, is this statement correct? That is why if we have excessive layers in a neural network, it will likely to learn an overly complicated function that fits perfectly on the training set only (overfitting). Could you please see if my personal understanding is accurate? $\endgroup$
    – YCCCCC
    Sep 2 '19 at 6:43
  • $\begingroup$ @YCCCCC yes, that's correct $\endgroup$
    – 1tan
    Sep 3 '19 at 11:31
  • $\begingroup$ Thanks you so much! $\endgroup$
    – YCCCCC
    Sep 14 '19 at 17:51
  • 1
    $\begingroup$ The answer is wrong. In general, without specifying certain activation functions and layer type, "Adding more and more layer" does not make the network learn "better and better", not in theory and also not in practice. Andrew Ng is very likely refering to a statement that was given in the ResNet paper. In their used architecture "adding more and more layers make the network learn better and better" is true. And in the context of ResNet it was shown that adding more layers does not help in practice. But the issue is NOT overfitting, but the solver.. $\endgroup$ Oct 3 '20 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.