When I add L2 regularization to my deep learning model the training and validation loss rate is increased.

Why ????


1 Answer 1


Suppose a neural network with a regular loss function.

$$ \sum_{i=1}^N L \left( y_i, \; \hat y_i \right) $$

Here, $y_i$ is label for the $i$-th example, while $\hat y_i$ is the model's prediction for the same. The loss function $L$ is a function that compares the actual and the predicted output and outputs a value indicating how close the prediction was to the actual output.

L2-regularization adds a norm penalty this loss function and as a result to each weight update.

$$ \sum_{i=1}^N L\left( y_i, \; \hat y_i \right) + λ \cdot \|W\|_2^2 $$

This penalty counters the actual update, meaning that it makes the weight updates harder. This has the effect of actually increasing the output your loss function.

What you should be looking for, by adding regularization to a model, isn't a reduction in training loss but a reduction in the validation loss. This would indicate that the regularization is successful in reducing your model's overfitting, which was its goal.

  • $\begingroup$ L2 increse the loss rate of both training and validation data. $\endgroup$
    – N.IT
    Commented Sep 13, 2018 at 1:13
  • $\begingroup$ Then maybe you are either regularizing too harshly or you didn't suffer from overfitting in the first place. Try reducing the regularization parameter ($λ$). $\endgroup$
    – Djib2011
    Commented Sep 13, 2018 at 7:16
  • 1
    $\begingroup$ Better $\LaTeX$ code for norms is \| rather than ||. $\endgroup$ Commented Sep 13, 2018 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.