0
$\begingroup$

I was watching Andrew Ng's video on ResNets, and he mentioned that "And in theory, as you make a neural network deeper, it should only do better and better on the training set." Here is my understanding of neural networks, as the model progresses through the model, the parameters it will learn will become more and more sophisticated, correct? Intuitively, it should be able to recognize/discover more detailed pattern information about the training set. Is my understanding correct?

Then why, in practice, does adding excessive layers to Neural Networks actually harm the performance of the model? Thanks in advance.

$\endgroup$
0
$\begingroup$

Adding more and more layer indeed make the network learn better and better on the training set. However, this causes a problem called "overfit". Overfitting means that your model works extremely well on training set but works poorly on validation set or testing set.

For your last question, the performance of Neural Network models is measured by the ability of the model to predict correctly for unseen data (or future data), which is the accuracy of predicting testing data. In practice, when you make your model deeper, the model will fit more completely to the training data. In this case, you increase the chance of getting an overfitted model. Therefore, the performance of the model (which is the accuracy of predicting testing data) will decrease due to overfitting into training data.

Reference: Overfitting

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Just to make sure, the model 'learns better and better' on the training set by casting layers of non-linearity activation functions, is this statement correct? That is why if we have excessive layers in a neural network, it will likely to learn an overly complicated function that fits perfectly on the training set only (overfitting). Could you please see if my personal understanding is accurate? $\endgroup$ – YCCCCC Sep 2 '19 at 6:43
  • $\begingroup$ @YCCCCC yes, that's correct $\endgroup$ – 1tan Sep 3 '19 at 11:31
  • $\begingroup$ Thanks you so much! $\endgroup$ – YCCCCC Sep 14 '19 at 17:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.