I've been training several auto encoders containing two GRUs as encoder and decoder during last year. It occurred to me that while training there were sequences of epochs which their loss did change just a little bit(less than 0.005 or so). Every time I took this as a sign that my model is not going to be better and stopped training. Now that I think about it I wonder what if I continued to train for more epochs and loss of model started to decrease again? Is there a factor to decide how long and how many epochs are enough to train recurrent networks?


3 Answers 3


Thats the whole point behind early stopping, don’t train the network to too small training error. It is always a bit tricky with early stopping but following framework might help.

When training, also output validation error

• Every time validation error improved, store a copy of the weights

• When validation error not improved for some time, stop

• Return the copy of the weights stored

After early stopping of the first run, train a second run and reuse validation data How to reuse validation data

  1. Start fresh, train with both training data and validation data up to the previous number of epochs

  2. Start from the weights in the first run, train with both training data and validation data until the validation loss < the training loss at the early stopping point

  • $\begingroup$ Choosing weights from early epochs witch validation loss is stopped improving is because the model is starting to overfit after that. Am I right? $\endgroup$ Jun 27, 2019 at 7:18
  • $\begingroup$ yes, you are right. $\endgroup$
    – emudria
    Jul 1, 2019 at 3:47

Generally we track loss on validation set during training. If the loss is not changing by a large factor for some number of epoch, we stop training.

Here the large factor and number of epoch are hyper parameters and you need to tune it according to the dataset.


Are you familiar with Dropout Layers? Dropout layers try to break dependencies between different Neurons by randomly leaving out Neurons, such that Neurons are (sometimes) encouraged to learn weights / patterns by themselves.

This is just a small description of a very good paper I would recommend you to read: Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Also in case you are using python, have a look at Tensorflow dropout respectively Keras dropout as they both also offer some sort of documentation as a further reading.

  • 1
    $\begingroup$ I'm familiar with drop out. It's a regularization method to prevent overfitting. But what is the relevance of it to know when to stop training? $\endgroup$ Jun 26, 2019 at 8:33
  • $\begingroup$ Yes, dropout does not tell you when to stop training. But on the other hand it somewhat solves this issue by preventing overfitting, and thus you could theoretically train as long as you want, as when proper configured, you will not suffer from overfitting. $\endgroup$
    – GrizZ
    Jun 26, 2019 at 8:45
  • $\begingroup$ @MarziehHeidari When rereading my answer, I must agree that it actually does not answer the question, as the main motivation for the question was to have some sort of termination constraint, whereas my answer tries to accomplish the same thing with a different approach. Must admit that the fingers were somewhat faster than the brain, sorry 'bout that. $\endgroup$
    – GrizZ
    Jun 26, 2019 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.