Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Are there any papers published which show differences of the regularization methods for neural networks, preferably on different domains (or at least different datasets)?
I am asking because I currently have the feeling that most people seem to use only dropout for regularization in computer vision. I would like to check if there would be a reason (not) to use different ways of regularization.
Two points:
Anyhow, I am a little skeptical of neural networks empirical studies. There are just too many hyperparameters to fine tune, from the topology of the network to the gradient descent optimization procedure to activation functions and whatever it is you are testing like regularization. Then, the entire thing is stochastic and usually performance gains are so small that you can hardly statistical test for differences. Many authors do not even bother doing statistical testing. They just average cross-validation and declare whatever model had the highest decimal point gain to be the winner.
You may find a study promoting dropout only to be contradicted by another promoting regularization.
I think it all boils down to aesthetics preferences. Dropout IMHO sounds more biological plausible than regularization. It also seems easier to calibrate. So, I personally prefer it when using a framework like TensorFlow. If we have to use our own neural network, which we often do, we will use regularization because it was easier to implement.
Definitely. The paper from the Creator himself, Geoffrey Hinton. https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf read it. But I encourage u to see the difference by yourself implementing it.
