In my CNN model, by using large number of epochs like 400 or above, the validations accuracy and some times test accuracy gets better, but I think this large number of epochs is not good idea? I am right or not? why?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
5
If your model is still improving (according to the validation loss), then more epochs are better. You can confirm this by using a hold-out test set to compare model checkpoints e.g. at epoch 100, 200, 400, 500.
Normally the amount of improvement reduces with time ("diminishing returns"), so it is common to stop once the curves is pretty-much flat, for example using EarlyStopping callback.
Different model requires different times to trains, depending on their size/architecture, and the dateset. Some examples of large models being trained on the ImageNet dataset (~1,000,000 labelled images of ~1000 classes):
- the original YOLO model trained in 160 epochs
- the ResNet model can be trained in 35 epoch
- fully-conneted DenseNet model trained in 300 epochs
The number of epochs you require will depend on the size of your model and the variation in your dataset.
The size of your model can be a rough proxy for the complexity that it is able to express (or learn). So a huge model can represent produce more nuanced models for datasets with higher diversity in the data, however would probably take longer to train i.e. more epochs.
Whilst training, I would recommend plotting the training and validation loss and keeping an eye on how they progress over epochs and also in relation to one another. You should of course expect both values to decrease, but you need to stop training once the lines start diverging - meaning that you are over-fitting to your specific dataset.
That is likely to happen if you train a large CNN for many epochs, and the graph could look something like this:
-
$\begingroup$ thank you for your detail answer, I train my model with 245, 294, and 343 epochs too, but the best accuracy achieved by 490 epochs, however, my data is CT scan images with size 480x480, the dataset size is 11,490 samples. $\endgroup$– HunarCommented Mar 2, 2019 at 14:17
-
$\begingroup$ @honar.cs - how big is your CNN model? How many parameters and how many layers? If 490 epochs gives you your best validation results, and you are sure you are not over-fitting, then I don't see a problem with just using that number of epochs :) $\endgroup$– n1k31t4Commented Mar 2, 2019 at 14:20
-
$\begingroup$ the model is consist of 5 conv layers, one dense layer with 256 nodes, but I do not compute the number of parameters because tensorflow does not compute it automatically, some times the gap between train and validation accuracy is big, but most of the times is in a good range. $\endgroup$– HunarCommented Mar 2, 2019 at 14:23
-
$\begingroup$ @honar.cs - if the gap in accuracy is less than about 5%, it might be ok. Given your have 5 layers and a single dense layer, 490 does sound fairly large. CT scan images are often complex, but 480x480 isn't too large (CT scans can be many times larger). If you want to improve your accuracy, I would suggest adding more layers and also thinking about adding regularisation through dropout layers (if you haven't already). $\endgroup$– n1k31t4Commented Mar 2, 2019 at 14:42
-
$\begingroup$ I have 2 regularization techniques which are dropout and L2, also the gap some times less than of %5 and other times more than, but I didn't try to add more layers. $\endgroup$– HunarCommented Mar 2, 2019 at 14:52
$\begingroup$
$\endgroup$
Assuming you track the performance with a validation set, as long as validation error is decreasing, more epochs are beneficial, model is improving on seen (training) and unseen (validation) data.
As soon as validation error starts to increase, it signals that model is over-fitting on training data, thus the learning process should be stopped. This method is independent of the details of a model.
In practice, save the model corresponding to the best validation error found so far, and stop the process only when validation error is consistently increasing. This can be done manually or with an ad-hoc threshold based on the distance of current validation error from the minimum error so far. This way, learning will not be stopped prematurely due to fluctuations in validation error.
$\begingroup$
$\endgroup$
It depends…
Smaller batch size will have increased performance on evaluation metrics (e.g., accuracy) because the model will make more frequent updates to the parameters.
Larger batch size will increase throughput (e.g., the number of images processed per second).
The trade-off often looks like a small drop in evaluation metrics with a large gain in throughput for increases in batch size. Thus if absolute performance on evaluation metrics is the only criteria, use small batches and take a long training time. If a dip in evaluation metrics is okay, you can greatly speed-up training by increasing batch size. For example, many applied models can be trained with batch sizes of 2048.
Overall, it is best to pick a multiple of 32 (32, 64, 256, 2048) in order to maximize the speed of data loading. The choice of which multiple of 32 often times depends on the architecture of the hardware that the model is training on.
It would be a good idea to empirically benchmark your model's performance, both evaluation metrics and training time, for different batch sizes.
answered Mar 2, 2019 at 18:12