does training for a large number of epochs lead to overfitting? I am concerned about this as I am getting an accuracy of nearly 1 on val and training dataset when I am training for 50 epochs
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Yes training for large number of epochs will lead to overfitting. This is because after a point the model starts learning noise from the training set. After a certain number of epochs, majority of what has to be learnt is already learnt and if you continue past that point, the noise present in the dataset starts affecting the model.
Based on your question, you think 50 epochs is not that many but how many epochs to set also depends on your dataset and what model you are using. If you have a large enough dataset, 50 epochs can be too much. Similarly if you have a small dataset, 50 epochs might not be enough.
On the same note, if you have a neural network with a lot of parameters, (example gpt2 or 3) you don't need that many epochs as the model is large and complex enough to learn from the data in just a few epochs. But if you have a relatively smaller neural network then you might need to increase the epochs so that the model can have sufficient iterations to learn from the data.
I would advise using learning curves to visualize how your model is performing for certain number of epochs. sklearn has a library learning_curves I think, for that purpose.
$\begingroup$
$\endgroup$
3
Training for 'large' number of epochs can indeed lead to overfitting. Accuracy of nearly 1 on train and validation set also tends me to believe that your model is overfitting. What is the accuracy on the test set? It should be quite close to the train & validation error if the model was trained properly. I suggest you take a look and try to employ some regularization techniques, maybe early stopping, dynamic learning rate, and also, some hyperparameter tuning.
That being said, the question lacks a lot of context. What kind of algorithm are you using? What is the machine learning problem/task that you are solving? What have you tried, which software, what arguments, etc. Try to give more details so that people can give you more specific advice.
-
$\begingroup$ I trained for about 30 epochs now , I am getting an accuracy of nearly 1 on test dataset as well , what could be causing this , I believe I should try feeding it a larger amount of data, any ideas as to what else could be going wrong $\endgroup$ May 13 at 19:10
-
$\begingroup$ If you did not cheat by committing any train-test contamination, you may have end up with a very good predictor. Is your task quite simple? It is quite hard to help without any information on the task, what does your data looks like, etc. $\endgroup$– CiodarMay 14 at 9:54
-
$\begingroup$ I am performing multiclass image classification into 4 categories , but I feel my dataset is quite small , just about 400 images , should I use more images? $\endgroup$ May 15 at 6:19