20
$\begingroup$

I am using a convolution neural network ,CNN. At a specific epoch, I only save the best CNN model weights based on improved validation accuracy over previous epochs.

Does increasing the number of epochs also increase over-fitting for CNNs and deep learning in general?

$\endgroup$

2 Answers 2

20
$\begingroup$

Yes, it may. In machine-learning there is an approach called early stop. In that approach you plot the error rate on training and validation data. The horizontal axis is the number of epochs and the vertical axis is the error rate. You should stop training when the error rate of validation data is minimum. Consequently if you increase the number of epochs, you will have an over-fitted model.

In deep-learning era, it is not so much customary to have early stop. There are different reasons for that but one of them is that deep-learning approaches need so much data and plotting the mentioned graph would be so much wavy because these approach use stochastic-gradient-like optimizations. In deep-learning again you may have an over-fitted model if you train so much on the training data. To deal with this problem, another approaches are used for avoiding the problem. Adding noise to different parts of models, like drop out or somehow batch normalization with a moderated batch size, help these learning algorithms not to over-fit even after so many epochs.

In general too many epochs may cause your model to over-fit the training data. It means that your model does not learn the data, it memorizes the data. You have to find the accuracy of validation data for each epoch or maybe iteration to investigate whether it over-fits or not.

$\endgroup$
10
  • 1
    $\begingroup$ @gamebm THe main reason we will have overfitting for a large number of epochs is that the weights will be large, and they will not have their initial small values. I mean this phenomenon is due to the fact that the weights will be biased towards values that are not mid values. $\endgroup$ Oct 24 at 6:29
  • 1
    $\begingroup$ @gamebm The point is that when we add terms like L1 to the cost function, it would be nice to find the global extremum because we consider regularisation in the cost function. The problem is that the cost is not convex for multi-layer networks. Consequently, we cannot find it. I mean in this case, it would be nice to find the global extremum, but we do not have any tool for that. $\endgroup$ Oct 26 at 6:08
  • 1
    $\begingroup$ @gamebm Sorry, I tried to consider solely your previous comment, not the entire content. For all approaches, whether you use regularisation or not, if you increase the number of epochs, the weights will be larger than the initial steps. If you add regularisation, the magnitude of the weights will be controlled, but the weights can still be large if you increase the number of epochs. On the other hand, the global minimum is not something that we can find for non-convex optimisation, the thing we have in deep learning. $\endgroup$ Oct 27 at 10:52
  • 1
    $\begingroup$ The other point is that I guess you're confused by two distinct terms. The global minimum does not have any relation to the number of epochs. If you have a simple regression problem with regularisation, you can have a cost function with a convex shape. So, you can find it without overfitting. I mean you can find the global min, and you may not have overfitting. Simply imagine some points that have linear behavior, and you're asked to fit a line. Overfitting depends on different aspects of the model as well. For the mentioned problem, if you use a multi-layer network, you'll not have $\endgroup$ Oct 27 at 10:56
  • 1
    $\begingroup$ a convex shape for your cost, and if you have an exaggerated regularisation to really avoid large weights, you will not have overfitting, but underfitting may be possible. I really tried to convey what I had in my mind :) $\endgroup$ Oct 27 at 10:58
2
$\begingroup$

YES. Increasing number of epochs over-fits the CNN model. This happens because of lack of train data or model is too complex with millions of parameters. To handle this situation the options are

  1. we need to come-up with a simple model with less number of parameters to learn
  2. add more data by augmentation
  3. add noise to dense or convolution layers
  4. add drop-out layers
  5. add l1 or l2 regularizers
  6. add early stopping
  7. check the model accuracy on validation data
  8. early stopping will tell you appropriate epochs without overfitting the model
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.