5
$\begingroup$

Why weights with large values cause neural networks to be overfitted, and consequently we use approaches like regularization to neutralize weights with large values?

$\endgroup$
5
$\begingroup$

Many strategies used in machine learning are explicitly designed to reduce the test error, possibly at the expense of increased training error. Generally, regularization is any modification we make to a learning algorithm that is intended to reduce its generalization error but not its training error.

The L2 parameter norm penalty commonly known as Weight decay is one of the simplest and most common kinds of regularization technique which forces the weights to become smaller, by adding a parameter norm $Ω(θ) = 1/2 || w||^{2}_2$ penalty to the objective function. For example, in linear regression, this gives us solutions that have a smaller slope, or put weight on fewer of the features. In other words, even though the model is capable of representing functions with much more complicated shape, weight decay has encouraged it to use a simpler function described by smaller coefficients.

Intuitively, in the feature space, only directions along which the parameters contribute significantly to reducing the objective function are preserved relatively intact. In directions that do not contribute to reducing the objective function, movement in this direction will not significantly increase the gradient.So, Components of the weight vector corresponding to such unimportant directions are decayed away through the use of the regularization throughout training.

Another simple explanation is when your weights are large, they are more sensitive to small noises in the input data. So, when a small noise is propagated through your network with large weights, it produces much different value in the output layer of the NN rather than a network with small weights.

Note that weight decay is not the only regularization technique. In the past few years, some other approaches have been introduced such as Dropout, Bagging, Early Stop, and Parameter Sharing which work very well in NNs.

There are other interesting findings in this very rich chapter.

$\endgroup$
  • $\begingroup$ The L2 parameter norm penalty commonly known as Weight decay is one of the simplest and most common kinds of regularization technique which forces the weights to become smaller, by adding a parameter Intuitively, in the feature space, only directions along which the parameters contribute significantly to reducing the objective function are preserved relatively intact would you please explain this more $\endgroup$ – Media Sep 26 '17 at 7:52
  • 1
    $\begingroup$ Consider gradient descent algorithm and gradient of the regularized objective function (L+λR). After adding the weight decay term (λR) learning algorithm reduces the weight vector by a constant factor on each step, before running gradient update. L2 regularization considers the input X as having higher variance, which makes it reduce the weights on features whose covariance with the output target is low compared to this added variance. $\endgroup$ – moh Sep 26 '17 at 14:47
  • 1
    $\begingroup$ Continue... Simply, if you have n weights in a layer of your NN, for each input X just a few of them (e.g. m weights) contribute significantly in producing target output and consequently making a decision in the output layer. So, L2 preserves these m weights intact and decades the remaining unimportant n-m weights (to prevent overfitting and also reduce sensitivity to small noise in the input). $\endgroup$ – moh Sep 26 '17 at 14:47
1
$\begingroup$

Large weights might correlate with certain patterns in the input data (x), this means that the model almost hard codes certain values. This then makes our traing data fit well but our test data fit less well.

You can also use dropout.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.