1
$\begingroup$

I know that L2 regularization technique to used to reduce over-fitting and penalize large weights. In more than one place I saw that it used like the code below in tensorflow library:

reg = tf.nn.l2_loss(w_conv1) + tf.nn.l2_loss(w_conv2) + \
      tf.nn.l2_loss(w_conv3) + tf.nn.l2_loss(w_conv4) + \
      tf.nn.l2_loss(w_conv5) + tf.nn.l2_loss(w_fc1) + \ 
      tf.nn.l2_loss(w_out)

    loss = tf.reduce_mean(loss + reg * beta)

I am confused about how this code penalize weights? the code adds L2 to weights then adding the result to loss. could anyone please explain this for me?

$\endgroup$
1
$\begingroup$

L2 loss is based on the square of the weights of your Network. As a given weight increases in size, the loss will increase exponentially (quadratically, to be precise). Neural Networks are therefore "pushed" to spread the weight values more evenly across each layer, since for such a quadratic loss factor it's better to have a lot of smaller weights, rather than few very big ones. The exponential relationship between weight value and loss is the reason why large weights get penalized. L2 loss is typically useful when your model suffers from overfitting, since weights redistribution across layers is good to prevent few neurons from staling all the "explanatory power" and generating overfitting.

(The main drawback of L2 loss is that the model becomes more sensitive to outliers, therefore less robust.)

Hope this helps, otherwise let me know.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ dear, your answer is useful but still now exactly answer my question, could you explain it on the written code? $\endgroup$ – Hunar A.Ahmed Jun 6 '19 at 19:59
  • $\begingroup$ Can you somehow back up your statement regarding outliers with literature? From my understanding $L_2$ represents a gaussian prior on weights and results in more equal weighting of features (in contrast to $L_1$) and therefore reduces sensitivity to outliers. I think you can find this statement in the Deep Learning book from Goodfellow I. $\endgroup$ – Andreas Look Jun 6 '19 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.