I have some excessive amount of data for the size of NN I am able to teach in a reasonable time.

If I feed all the data into the network it stops learning at some point and a resulting model shows all signs of being overfit.

Intuitively if I increase dropout prob the model should learn less aggressively from data and gain from more data being fed into it.

Is my logic sound?

  • 1
    $\begingroup$ This seems easy to try. Have you tried it? What happened? $\endgroup$ – Neil Slater Jul 11 '17 at 7:13
  • $\begingroup$ @NeilSlater Doing it now. Higher dropout prob means longer training time. I will update once training results become available to me. $\endgroup$ – Denis Kulagin Jul 11 '17 at 8:11
  • $\begingroup$ to reduce overfit, in addition to dropout you could also use l1 or l2 weight regularization. Also Batch Normalization layer provides a small degree of regularization and reduces reliance on dropout: arxiv.org/pdf/1502.03167.pdf $\endgroup$ – Vadim Smolyakov Aug 16 '17 at 3:57

You are right, increasing the dropout proportion will help.

However, this looks like a setting where early stopping will be a very good choice. Although dropout, weight decay and batch-norm can work propperly, the fact that you easily overfit your training set would make it an appropiate scenario to try early stopping.

In addition, as the neural network takes very short to be trained you can train many of them (on some subset of the training set, making them weak learners) and create an ensemble to make the final predictions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.