L2 regularization increase the loss rate of the deep learning model

When I add L2 regularization to my deep learning model the training and validation loss rate is increased.

Why ????

Suppose a neural network with a regular loss function.

$$\sum_{i=1}^N L \left( y_i, \; \hat y_i \right)$$

Here, $y_i$ is label for the $i$-th example, while $\hat y_i$ is the model's prediction for the same. The loss function $L$ is a function that compares the actual and the predicted output and outputs a value indicating how close the prediction was to the actual output.

L2-regularization adds a norm penalty this loss function and as a result to each weight update.

$$\sum_{i=1}^N L\left( y_i, \; \hat y_i \right) + λ \cdot \|W\|_2^2$$

This penalty counters the actual update, meaning that it makes the weight updates harder. This has the effect of actually increasing the output your loss function.

What you should be looking for, by adding regularization to a model, isn't a reduction in training loss but a reduction in the validation loss. This would indicate that the regularization is successful in reducing your model's overfitting, which was its goal.

• L2 increse the loss rate of both training and validation data.
– N.IT
Sep 13 '18 at 1:13
• Then maybe you are either regularizing too harshly or you didn't suffer from overfitting in the first place. Try reducing the regularization parameter ($λ$). Sep 13 '18 at 7:16
• Better $\LaTeX$ code for norms is \| rather than ||. Sep 13 '18 at 13:16