2
$\begingroup$

I am trying to train an AlexNet image model on the RVL-CDIP Dataset. The dataset consists of 320,000 training images, 40,000 validation images, and 40,000 test images.

Since the dataset is huge I started training on 500 (per class) sample from the training set. The result is below: training on 500 (per class) sample, 32 batches , 40 epochs

we can see from the graph above that the validation loss started decreasing at a much slower rate around epoch 20 while training loss continued decreasing the same. This means our model started overfitting the data? I assume that this is probably because the data i have in the training set is not enough to get better results on the validation set? (validation data is also a 500 (per class) sample from the whole validation set)

is it a correct approach to train the model on a small sample (eg. 500 images per class), save the model, load the saved model weights and then train again with a larger sample (eg 1000 images)? My intution is that this way the model would have new data every new run that helps it to learn more about the validation set. And if this approach is correct, when training the model for the second time with a larger sample, should the training sample include images (some or all) that were trained in the first model?

You can find the full code with results here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It reminds me of this question, the training loss is decreasing faster than the validation loss. I understand there is some overfitting, as the model is learning some patterns that are only in the training set, but the model is still learning some patterns that are more general, as the validation loss is decreasing as well. To me it would be more of an issue if the validation loss increased, but it is not the case.

Edit

Usually neural networks are trained with all the data, training by using mini-batch gradient descent already does what you mention in your approach without the need of storing the model in memory. So, I would train with as much data as possible, to have a model with the least possible variance. If you are not feeding the data using generators and the whole dataset doesn't fit into memory, I recommend to use them, or train with a model which is as big as possible given your memory limitations.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I saved the model after 40 epochs and tried two experiments as follows, First, trained the model further with the same exact setup, validation loss stays constant after some epochs around 1.4 while training loss keeps decreasing the same as before. Second, trained the model further but this time I added 500 images to the training sample (so 1000 in total). Validation loss also stays constant after some epochs. couldn't this mean that my training sample (500 & 1000 out of around 20000 per class) is not enough to generalize over the validation set? $\endgroup$
    – Omar Rayyan
    Jul 20, 2020 at 10:02
  • $\begingroup$ I don't think so, I think batch size is usually chosen in terms of computational performance, not as a hyperparameter to tune - there are hyperparameters that generally impact more on the network. It might be that in your case something's happening, but I don't see it as a general thing, and regarding the interpretation on the generalization, I've never heard about it before. In fact stochastic gradient descent usually works well, but it is slow $\endgroup$
    – David Masip
    Jul 20, 2020 at 10:19
  • $\begingroup$ I am not sure if I understand correctly, but here I am not changing the batch size itself, the batch size stays 32 in all cases. what I am doing is increasing the number of training instances I feed to the model as a whole gradually (while batch size stays 32). $\endgroup$
    – Omar Rayyan
    Jul 20, 2020 at 11:51
  • $\begingroup$ alright, good! didn't understand, my bad $\endgroup$
    – David Masip
    Jul 20, 2020 at 12:48
  • $\begingroup$ I've updated my answer accordingly $\endgroup$
    – David Masip
    Jul 20, 2020 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.