9
$\begingroup$

Are there any papers published which show differences of the regularization methods for neural networks, preferably on different domains (or at least different datasets)?

I am asking because I currently have the feeling that most people seem to use only dropout for regularization in computer vision. I would like to check if there would be a reason (not) to use different ways of regularization.

$\endgroup$
3
$\begingroup$

Two points:

  1. Dropout is also usually compared with neural networks ensembles. It seems it has some of the performance benefits of training and averaging several neural networks.
  2. Dropout is easier to calibrate than regularization. There is only one hyperparameter which is the dropout rate and people widely use 0.5 while training (and then 1.0 on evaluation of course :)), see e.g. this TensorFlow example.

Anyhow, I am a little skeptical of neural networks empirical studies. There are just too many hyperparameters to fine tune, from the topology of the network to the gradient descent optimization procedure to activation functions and whatever it is you are testing like regularization. Then, the entire thing is stochastic and usually performance gains are so small that you can hardly statistical test for differences. Many authors do not even bother doing statistical testing. They just average cross-validation and declare whatever model had the highest decimal point gain to be the winner.

You may find a study promoting dropout only to be contradicted by another promoting regularization.

I think it all boils down to aesthetics preferences. Dropout IMHO sounds more biological plausible than regularization. It also seems easier to calibrate. So, I personally prefer it when using a framework like TensorFlow. If we have to use our own neural network, which we often do, we will use regularization because it was easier to implement.

$\endgroup$
0
$\begingroup$

Definitely. The paper from the Creator himself, Geoffrey Hinton. https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf read it. But I encourage u to see the difference by yourself implementing it.

$\endgroup$
  • 2
    $\begingroup$ The paper doesn't explicitly compare between different regularisation approaches, except by demonstrating dropout as improving on state-of-the-art results at the time (previous results most likely did use some other forms of regularisation, but they are not listed). It also does mention maxnorm weight constraints as an effective additional regulariser to augment dropout. $\endgroup$ – Neil Slater Apr 8 '16 at 8:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.