2
$\begingroup$

I'm building a RNN (recurrent neural network) with LSTM cells. I'm using time series to perform anomaly detection.

When training my RNN I'm using a dropout of 0.5 and I'm early stopping with a patience of 5 epochs when my validation loss is increasing.

Does it make sense to use early stopping in combination with dropout?

$\endgroup$
1
$\begingroup$

It does make sense, they are just two different things.

Dropout only makes your model learning harder, and by this it helps the parameters of the model act in different ways and detect different features, but even with dropout you can potentially overfit your traning set.

On the other hand, early stopping prevents your model from overfitting by taking the best model on your validation data so far.

However, for the sake of simplicity, I think it is easier to just use dropout (training a neural network is not easy and the training may not be successful due to many different reasons, it is a good practice to reduce the possible reasons why the training is failing as much as possible). Unless you have short time to train your network, with a sufficiently high amount of dropout you will ensure that your model is not overfitting.

My final recommendation is: just use dropout. If using a 0.5 dropout rate still overfits, set a higher dropout rate.

$\endgroup$
  • $\begingroup$ I think you recommendation isn't substantiated... $\endgroup$ – Mike Evers Apr 20 '18 at 9:13
  • $\begingroup$ It might not be, but Andrew Ng does not recommend early stopping. This is because it is a good practice to make a difference between training and validation, and early stopping is kind of inbetween. If you have trained neural nets you know what I am talking about regarding the problems that may arise, and dropout is applied successfully in many cases. I have never had an overfitting problem when using dropout with a sufficiently large dropout rate. $\endgroup$ – David Masip Apr 20 '18 at 9:17
  • $\begingroup$ I did a little research, and on Hinton's paper arxiv.org/abs/1207.0580, he claims that (dropout) 'It allows much larger nets to be trained and removes the need for early stopping'. Is this more substantiated? $\endgroup$ – David Masip Apr 20 '18 at 13:41
  • 1
    $\begingroup$ Early stopping is a hack. If you do not want to lose much time tweaking your regularization to avoid overfitting, then go ahead and use early stopping. $\endgroup$ – Ricardo Cruz Apr 20 '18 at 14:08
1
$\begingroup$

Avoid early stopping and stick with dropout. Andrew Ng does not recommend early stopping in one of his courses on orgothonalization [1] and the reason is as follows. For a typical machine learning project, we have the following chain of assumptions for our model:

Fit the training set well on the cost function

Fit the dev set well on the cost function

Fit the test set well on the cost function

Performs well in the real world

And what we want are tools that can target one of these four objectives and not the others in order to keep improving our models more efficiently, and this concept is called "orthogonalization." An analogy would be the different options in photo editing apps such as brightness, contrast, and saturation adjustments, which are independent of, or "orthogonal" to one another. Examples of orthogonal optimization strategies for the four goals are listed below:

Fit the training set well on the cost function (e.g. bigger neural network; Adam optimization)

Fit the dev set well on the cost function (e.g. regularization; bigger training set)

Fit the test set well on the cost function (e.g. bigger dev set)

Performs well in the real world (e.g. change the test set; change the cost function)

Because early stopping both fits the training set less well and improves the dev set performance at the same time, it is not orthogonal and Ng advises us not to use it.


Reference: [1] Week 1 of Course 3: Structuring Machine Learning Projects of Coursera Deep Learning Specialization

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.