3
$\begingroup$

I have been building a neural network for classification.

To select my best model. I have been using 10-Fold cross validation. and selected the network that gives the highest mean accuracy.

Now that I have selected the best model, I want to use all the data I have to train this model because the amount of data I have is limited (I will merge training, dev and test data).

My issue is that, when training with all the data, I don't know when to stop training. Training loss is not an indicator for sure. Usually, I have a development set that I use to monitor training. When the training loss does not improve anymore, I stop training.

Any suggestions on how to supervise a model with only training data? In other words, how to tell when the network needs to stop?

Improve this question
$\endgroup$
3
  • $\begingroup$ Relevant cross question: stats.stackexchange.com/a/26535/67965 $\endgroup$
    – E_net4
    Sep 14, 2017 at 8:56
  • $\begingroup$ Were you using early stopping or fixed number of epochs in your cross-validation? $\endgroup$
    – Neil Slater
    Sep 14, 2017 at 9:00
  • $\begingroup$ I used two setting. One where I stop when the dev loss is not improving and one where I run for 150 regardless of the dev loss. Both approaches led to the same model. The thing when I use 150 epochs is that in the last epoch, I am not getting the best accuracy for any given fold. Different folds reach the best accuracy at different epochs. Hence, my confusion when using the whole training data $\endgroup$
    – ryuzakinho
    Sep 14, 2017 at 9:06

1 Answer 1

Reset to default
4
$\begingroup$

When 'overtraining' is not a problem (as in it will not diverge if you use more time), just use all your data and the empirically found optimal hyper parameters. In case of neural networks this is not the case (although in my experience, a lot of architectures converge to a specific test error, and take a long time to end up diverging again). I see a few options that you could try:

  • Most obvious one is keeping a (small) validation set around to use as indicator for early stopping (don't think of this as throwing away data, you still use it to train your network better)
  • Use same weight initalization as one of your folds and run for the same amount of epochs, same initialization should make convergence rate more similar than new random initialization
  • Keep all the cross validation models and use them in an ensemble instead of retraining the full model
Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks @Jan. I have thought of the last option but it is computationally expensive. Now, regarding the first option you suggest, I am not sure to understand at what point the dev data will be used. Do you mean that my dev data should be a sub-set of the training data this is why you said it would not be thrown away? $\endgroup$
    – ryuzakinho
    Sep 14, 2017 at 9:03
  • $\begingroup$ For early stopping you need some validation set that measures generalization, which would be similar to a fold in your CV scheme. Once you start losing performance on this set you need to stop training. You would not use this data for backpropagation, which makes it feel like you are not using this data, but this is not the case because you use it to measure convergence and prevent divergence $\endgroup$
    – Jan van der Vegt
    Sep 14, 2017 at 9:34
  • $\begingroup$ I understand better now! One question remains: how to choose the dev set? When I did CV, I got results in the range 65-85% of accuracy.This means that depending on the content of the dev set, the supervision would be different ! $\endgroup$
    – ryuzakinho
    Sep 14, 2017 at 10:01
  • $\begingroup$ That depends strongly on your problem instance. You could just randomly sample from your set, but if there are time based dependencies this could be somewhat bad. But since it is basically for model selection and not for reporting in your paper or to your manager/business contact I don't think it matters too much. The size is another issue, too small means high variance estimates and too large means not enough data for the NN optimizer. If 10% folds had high variance in accuracy maybe you need a bit more $\endgroup$
    – Jan van der Vegt
    Sep 14, 2017 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.