4
$\begingroup$

Regularization is used to decrease the capacity of a machine learning model to avoid overfitting. Why don't we just use a model with less capacity (e.g. decrease the number of layers). This would also benefit the computational time and memory.

My guess would be that different regularization methods make different assumptions of the dataset. If so, what assumptions are made for the common regularizations (L1, L2, dropout, any other)

Thanks in advance!

$\endgroup$
4
$\begingroup$

Regularization does decrease the capacity of the model in some sense, but as you already guessed, different capacity reductions result in models of different quality and are not interchangeable.

L1 can be interpreted as making the assumption that the influence of different factors (represented by neurons) on each other shouldn’t be assumed without significant support by data (i.e. the gain achieved by larger influence has to outweight the L1 loss associated with increased absolute value of the parameter that „connects“ them).

L2 does the same, but makes this dependent on the connection strength, i.e. very light connections basically need no support (and are therefore not driven further to exact zero) and very large connections are almost impossible.

Dropout can be interpreted as training a large amount of smaller networks and using the approximated average network for inference: „So training a neural network with dropout can be seen as training a collection of 2^n thinned networks with extensive weight sharing, where each thinned network gets trained very rarely, if at all.“ Dropout: A Simple Way to Prevent Neural Networks from Overfitting

All these methods make certain network parameter combinations highly improbable or even impossible to achieve for a given dataset, which otherwise could have been the result of the training. In this sense, the capacity of the model is reduced. But as one can imagine, some capacity reductions are more useful than others.

|improve this answer|||||
$\endgroup$
1
$\begingroup$

Regularization is not primarily used to avoid overfitting. Regularization shrinks weights which are not „useful“ to make good predictions. And regularization is also used in many other models, where it has more the notion of feature or model selection (regression, logit, boosting).

The benefit of regularization is, that you can work with a model which has high capacity, but using regularization you don‘t need to worry too much about features (and their representation in NN). Regularization kind of automatically drops weights which are not too important. So it is a really useful tool, e.g. in cases where you have a lot of information but you don‘t know what information is actually needed to make good predictions.

Dropout is a different thing, since it means to randomly drop weights. Shrinking means that weights which do not contribute much to good predictions, receive less attention by the model. L1 can shrink weights to zero, while L2 will never be exactly zero.

To learn more about regularization, you may look at Introduction to Statistical Learning. In the book, there is a really instructive Chapter on the issue.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Regularization is primarily used to avoid overfitting. It discourages more complex models in favor of simpler ones that explain the data (almost) as well. In your answer you motivate regularization just by dropping weights, but what is the motivation for dropping irrelevant weights? It's the avoidance of overfitting. $\endgroup$ – leonard Aug 9 '19 at 9:46
  • $\begingroup$ When you look at the Lasso for instance, you see that it has a more general purpose, namely model selection which - among other things - can be used to fight overfitting. However, the purpose is a more general one, e.g. to also work with high dimensional date. Stating that regularization only targets overfitting simplifies the approach too much. $\endgroup$ – Peter Aug 9 '19 at 10:32
  • $\begingroup$ rss.onlinelibrary.wiley.com/doi/abs/10.1111/… $\endgroup$ – Peter Aug 9 '19 at 10:32
  • $\begingroup$ By introducing regularization you accept a slightly higher training error (otherwise introducing it wouldn’t have effects) in the hope of getting a better generalization, i.e. lower test error. This is what is called avoiding overfitting. Of course, there are other potential benefits for some of the regularization approaches, like e.g. easier explainability with L1, but saying that regularization‘s primary purpose is not overfitting avoidance, is wrong in my opinion. $\endgroup$ – leonard Aug 9 '19 at 12:04
  • $\begingroup$ Btw: Even if you say Dropout is something different, it’s still regularization just as L1 and L2 and other regularization approaches as e.g. data augmentation. One might also consider feature selection as regularization. Each approach has side effects and benefits, but the one thing in common is fighting overfitting. $\endgroup$ – leonard Aug 9 '19 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.