I see some literature consider L2 loss (least squared error) and mean squared error loss are two different kinds of loss functions.

However, it seems to me these two loss functions essentially compute the same thing (with a 1/n factor difference).

So I am wondering if I have missed anything? Is there any scenario that one should use one of the two loss functions?

  • 2
    $\begingroup$ Could you provide a reference to a source where the two losses are considered to be different? $\endgroup$ – Neil Slater Jan 1 '18 at 8:41
  • $\begingroup$ @Edamame are you sure least squared error is called L2? $\endgroup$ – Media Jan 5 '18 at 14:24

Function $L_2(x):=\left \|x \right \|_2$ is a norm, it is not a loss by itself. It is called a "loss" when it is used in a loss function to measure a distance between two vectors, $\left \| y_1 - y_2 \right \|^2_2$, or to measure the size of a vector, $\left \| \theta \right \|^2_2$. This goes with a loss minimization that tries to bring these quantities to the "least" possible value.

These are some illustrations:

  1. $L_p$ norm: $L_p(x) := \left \|x \right \|_p = (\sum_{i=1}^{D} |x_i|^p)^{1/p}$,
    where $D$ is the dimension of vector $x$,

  2. Squared error: $\mbox{SE}(A, \theta) =\sum_{n=1}^{N} \left \| y_n - f_{\theta}(x_n) \right \|^2_2$,
    where $A=\{(x_n, y_n)_{n=1}^{N}\}$ is a set of data points, and $f_{\theta}(x_n)$ is model's estimation of $y_n$,

  3. Mean squared error: $\mbox{MSE}(A, \theta) =\mbox{SE}(A, \theta)/N$,

  4. Least squares optimization: $\theta^*=\mbox{argmin}_{\theta} \mbox{MSE}(A, \theta)$$=\mbox{argmin}_{\theta} \mbox{SE}(A, \theta)$,

  5. Ridge loss: $\mbox{R}(A, \theta, \lambda) = \mbox{MSE}(A, \theta) + \lambda\left \| \theta \right \|^2_2$

  6. Ridge optimization (regression): $\theta^*=\mbox{argmin}_{\theta} \mbox{R}(A, \theta, \lambda)$.

In all of the above examples, $L_2$ norm can be replaced with $L_1$ norm or $L_\infty$ norm, etc.. However the names "squared error", "least squares", and "Ridge" are reserved for $L_2$ norm. For example for $L_1$, "squared error" becomes "absolute error":

  1. Absolute error: $\mbox{AE}(A, \theta) =\sum_{n=1}^{N} \left \| y_n - f_{\theta}(x_n) \right \|_1$,
| improve this answer | |

To be precise, L2 norm of the error vector is a root mean-squared error, up to a constant factor. Hence the squared L2-norm notation $\|e\|^2_2$, commonly found in loss functions.

However, $L_p$-norm losses should not be confused with regularizes. For instance, a combination of the L2 error with the L2 norm of the weights (both squared, of course) gives you a well known ridge regression loss, while a combination of L2 error + L1 norm of the weights gives rise to a Lasso regression.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.