11
$\begingroup$

When training neural networks, there are at least 4 ways to regularize the network:

  • L1 Regularization
  • L2 Regularization
  • Dropout
  • Batch Normalization

    plus of course other things like weight sharing and reducing the number of connections, which might not be regularization in the strictest sense.

    But how would one choose which of those regularization methods to use? Is there a more principled way than "just try everything and see what works"?

  • $\endgroup$
    2
    • 3
      $\begingroup$ Do neural networks have principles? The principle for black box methods is to try everything and see what works $\endgroup$ May 25, 2016 at 7:38
    • $\begingroup$ And this is quite sad, don't you find? $\endgroup$
      – Alex
      Oct 26, 2017 at 9:59

    4 Answers 4

    8
    $\begingroup$

    There are not any strong, well-documented principles to help you decide between types of regularisation in neural networks. You can even combine regularisation techniques, you don't have to choose just one.

    A workable approach can be based on experience, and following literature and other people's results to see what gave good results in different problem domains. Bearing this in mind, dropout has proved very successful for a broad range of problems, and you can probably consider it a good first choice almost regardless of what you are attempting.

    Also sometimes just picking a option you are familiar with can help - working with techniques you understand and have experience with may get you better results than trying a whole grab bag of different options where you are not sure what order of magnitude to try for a parameter. A key issue is that the techniques can interplay with other network parameters - for instance, you may want to increase size of layers with dropout depending on the dropout percentage.

    Finally, it may not matter hugely which regularisation techniques you are using, just that you understand your problem and model well enough to spot when it is overfitting and could do with more regularisation. Or vice-versa, spot when it is underfitting and that you should scale back the regularisation.

    $\endgroup$
    3
    $\begingroup$

    Method of regularization

    For the following 4 techniques, L1 Regularization and L2 Regularization are needless to say that they must be a method of regularization. They shrink the weight. L1 would concentrate on shrinking a smaller amount of weight if the weights have higher importance.

    Dropout prevents overfitting by temporarily dropping out neurons. Eventually, it calculates all weights as an average so that the weight won't be too large for a particular neuron and hence it is a method of regularization.

    Batch Normalization should not be a method of regularization because the main purpose of it is to speed up the training by selecting a batch and forcing the weight to be distributed near 0, not too large, not too small.

    Choosing it

    For me, mini-batch is a must because it can speed up the process and improve the performance of network every time.

    L1 and L2 are both similar and I would prefer L1 in small network.

    Ideally, dropout should apply if there is a large variation problem or overfitting.

    Last but not the least, I agree with Neil Slater that it depends on the situation and there will never be an optimum solution.

    I recommend you to read this for further information. This is a very good material. http://neuralnetworksanddeeplearning.com/chap3.html

    $\endgroup$
    0
    $\begingroup$

    L1 Regularization: Lasso Regression - works well for feature selection in case we have a huge number of features. - as it makes extensive feature selection shrinking

    L2 Regularization: Ridge Regression - perceive features with higher variance & shrink features with lower variance, - it regularizes biased values

    Dropout as a form of regularisation, is adding robustness

    Batch Normalization - "normalization of the layers' inputs by re-centering and re-scaling" - recommended after each Dense-layer

    p.s. here

    Another method is to alter the labels of the model in a specific way, which is a method of regularization

    • I assume is smth like correction for DV , as well it's possible to make corrections for IVs (on the base of previous expectations) -- for Sequence analysis (not random vars)
    $\endgroup$
    -1
    $\begingroup$

    Look at those algorithmic choices as additional hyperparameters and optimize them the same way as you do for your other hyperparameters. Typically this will require more data though.

    $\endgroup$
    1
    • 2
      $\begingroup$ Hi Alex, Welcome to DS.SE. This is a Q&A site that has the richest answers floating to the top via voting. Someone has voted you down, perhaps since your answer is quite short and generally explaining the solution (e.g.) isnt explaining the details of hyperparameters, a term which wasnt used by the original poster. $\endgroup$
      – Marcus D
      May 25, 2016 at 14:00

    Your Answer

    By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

    Not the answer you're looking for? Browse other questions tagged or ask your own question.