I am trying to train an AlexNet image model on the RVL-CDIP Dataset. The dataset consists of 320,000 training images, 40,000 validation images, and 40,000 test images.
we can see from the graph above that the validation loss started decreasing at a much slower rate around epoch 20 while training loss continued decreasing the same. This means our model started overfitting the data? I assume that this is probably because the data i have in the training set is not enough to get better results on the validation set? (validation data is also a 500 (per class) sample from the whole validation set)
is it a correct approach to train the model on a small sample (eg. 500 images per class), save the model, load the saved model weights and then train again with a larger sample (eg 1000 images)? My intution is that this way the model would have new data every new run that helps it to learn more about the validation set. And if this approach is correct, when training the model for the second time with a larger sample, should the training sample include images (some or all) that were trained in the first model?
You can find the full code with results here