Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
When training neural networks, there are at least 4 ways to regularize the network:
plus of course other things like weight sharing and reducing the number of connections, which might not be regularization in the strictest sense.
But how would one choose which of those regularization methods to use? Is there a more principled way than "just try everything and see what works"?
There are not any strong, well-documented principles to help you decide between types of regularisation in neural networks. You can even combine regularisation techniques, you don't have to choose just one.
A workable approach can be based on experience, and following literature and other people's results to see what gave good results in different problem domains. Bearing this in mind, dropout has proved very successful for a broad range of problems, and you can probably consider it a good first choice almost regardless of what you are attempting.
Also sometimes just picking a option you are familiar with can help - working with techniques you understand and have experience with may get you better results than trying a whole grab bag of different options where you are not sure what order of magnitude to try for a parameter. A key issue is that the techniques can interplay with other network parameters - for instance, you may want to increase size of layers with dropout depending on the dropout percentage.
Finally, it may not matter hugely which regularisation techniques you are using, just that you understand your problem and model well enough to spot when it is overfitting and could do with more regularisation. Or vice-versa, spot when it is underfitting and that you should scale back the regularisation.
Method of regularization
For the following 4 techniques, L1 Regularization and L2 Regularization are needless to say that they must be a method of regularization. They shrink the weight. L1 would concentrate on shrinking a smaller amount of weight if the weights have higher importance.
Dropout prevents overfitting by temporarily dropping out neurons. Eventually, it calculates all weights as an average so that the weight won't be too large for a particular neuron and hence it is a method of regularization.
Batch Normalization should not be a method of regularization because the main purpose of it is to speed up the training by selecting a batch and forcing the weight to be distributed near 0, not too large, not too small.
Choosing it
For me, mini-batch is a must because it can speed up the process and improve the performance of network every time.
L1 and L2 are both similar and I would prefer L1 in small network.
Ideally, dropout should apply if there is a large variation problem or overfitting.
Last but not the least, I agree with Neil Slater that it depends on the situation and there will never be an optimum solution.
I recommend you to read this for further information. This is a very good material. http://neuralnetworksanddeeplearning.com/chap3.html
L1 Regularization: Lasso Regression - works well for feature selection in case we have a huge number of features. - as it makes extensive feature selection shrinking
L2 Regularization: Ridge Regression - perceive features with higher variance & shrink features with lower variance, - it regularizes biased values
Dropout as a form of regularisation, is adding robustness
Batch Normalization - "normalization of the layers' inputs by re-centering and re-scaling" - recommended after each Dense-layer
p.s. here
Another method is to alter the labels of the model in a specific way, which is a method of regularization
Look at those algorithmic choices as additional hyperparameters and optimize them the same way as you do for your other hyperparameters. Typically this will require more data though.
