I've been training several auto encoders containing two GRUs as encoder and decoder during last year. It occurred to me that while training there were sequences of epochs which their loss did change just a little bit(less than 0.005 or so). Every time I took this as a sign that my model is not going to be better and stopped training. Now that I think about it I wonder what if I continued to train for more epochs and loss of model started to decrease again? Is there a factor to decide how long and how many epochs are enough to train recurrent networks?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
Thats the whole point behind early stopping, don’t train the network to too small training error. It is always a bit tricky with early stopping but following framework might help.
When training, also output validation error
• Every time validation error improved, store a copy of the weights
• When validation error not improved for some time, stop
• Return the copy of the weights stored
After early stopping of the first run, train a second run and reuse validation data How to reuse validation data
Start fresh, train with both training data and validation data up to the previous number of epochs
Start from the weights in the first run, train with both training data and validation data until the validation loss < the training loss at the early stopping point
-
$\begingroup$ Choosing weights from early epochs witch validation loss is stopped improving is because the model is starting to overfit after that. Am I right? $\endgroup$ Jun 27, 2019 at 7:18
-
$\begingroup$
$\endgroup$
Generally we track loss on validation set during training. If the loss is not changing by a large factor for some number of epoch, we stop training.
Here the large factor and number of epoch are hyper parameters and you need to tune it according to the dataset.
$\begingroup$
$\endgroup$
3
Are you familiar with Dropout Layers?
Dropout layers try to break dependencies between different Neurons by randomly leaving out Neurons, such that Neurons are (sometimes) encouraged to learn weights / patterns by themselves.
This is just a small description of a very good paper I would recommend you to read: Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Also in case you are using python, have a look at Tensorflow dropout respectively Keras dropout as they both also offer some sort of documentation as a further reading.
-
1$\begingroup$ I'm familiar with drop out. It's a regularization method to prevent overfitting. But what is the relevance of it to know when to stop training? $\endgroup$ Jun 26, 2019 at 8:33
-
$\begingroup$ Yes, dropout does not tell you when to stop training. But on the other hand it somewhat solves this issue by preventing overfitting, and thus you could theoretically train as long as you want, as when proper configured, you will not suffer from overfitting. $\endgroup$– GrizZJun 26, 2019 at 8:45
-
$\begingroup$ @MarziehHeidari When rereading my answer, I must agree that it actually does not answer the question, as the main motivation for the question was to have some sort of termination constraint, whereas my answer tries to accomplish the same thing with a different approach. Must admit that the fingers were somewhat faster than the brain, sorry 'bout that. $\endgroup$– GrizZJun 26, 2019 at 9:28