When I add L2 regularization to my deep learning model the training and validation loss rate is increased.

Why ????


Suppose a neural network with a regular loss function.

$$ \sum_{i=1}^N L \left( y_i, \; \hat y_i \right) $$

Here, $y_i$ is label for the $i$-th example, while $\hat y_i$ is the model's prediction for the same. The loss function $L$ is a function that compares the actual and the predicted output and outputs a value indicating how close the prediction was to the actual output.

L2-regularization adds a norm penalty this loss function and as a result to each weight update.

$$ \sum_{i=1}^N L\left( y_i, \; \hat y_i \right) + λ \cdot \|W\|_2^2 $$

This penalty counters the actual update, meaning that it makes the weight updates harder. This has the effect of actually increasing the output your loss function.

What you should be looking for, by adding regularization to a model, isn't a reduction in training loss but a reduction in the validation loss. This would indicate that the regularization is successful in reducing your model's overfitting, which was its goal.

  • $\begingroup$ L2 increse the loss rate of both training and validation data. $\endgroup$
    – N.IT
    Sep 13 '18 at 1:13
  • $\begingroup$ Then maybe you are either regularizing too harshly or you didn't suffer from overfitting in the first place. Try reducing the regularization parameter ($λ$). $\endgroup$
    – Djib2011
    Sep 13 '18 at 7:16
  • 1
    $\begingroup$ Better $\LaTeX$ code for norms is \| rather than ||. $\endgroup$ Sep 13 '18 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.