1
$\begingroup$

I'm reading a paper published in nips 2021. There's a part in it that is confusing:

This loss term is the mean squared error of the normalized feature vectors and can be written as what follows: enter image description here Where $\left\|.\right\| _2$is $\ell_2$ normalization,$\langle , \rangle$ is the dot product operation.

As far as I know MSE loss function looks like : $L=\frac{1}{2}(y - \hat{y})^{2}$

How does the above equation qualify as an MSE loss function?

$\endgroup$

1 Answer 1

2
$\begingroup$

Recall what mean square error is actually measuring... the Euclidean distance between some regressed function, $\hat y$ and the true signal/function $y$ evaluated at every input $x$. The above is a more formalized vector definition, but is still very much the same.

Starting from this idea that the Euclidean distance is coming into play:

$ d(f_{1}(x),f_{2}(x))^{2} = \langle f_{1}(x) - f_{2}(x), f_{1}(x) - f_{2}(x) \rangle = \langle f_{1}(x),f_{1}(x) \rangle + \langle f_{2}(x),f_{2}(x) \rangle - 2 \langle f_{1}(x),f_{2}(x) \rangle = 2 (1 - \langle f_{1}(x),f_{2}(x) \rangle) = 2 - 2 \langle f_{1}(x),f_{2}(x) \rangle$.

The denominator is just to make each vector (and by extension, their dot product) of unit length.

Hope this helps!

$\endgroup$
3
  • $\begingroup$ You reduced the equation based on the equality with one of the feature norm, which is fine, it was given in the problem. However, the formula in OP questions adds division by norms, which are also equal with $1$. IMHO this is very misleading. $\endgroup$
    – rapaio
    Feb 12, 2022 at 13:25
  • $\begingroup$ there was a little hand-waving here since the OP was more about content than rigor, but it isn't clear how the above is misleading? $\endgroup$ Feb 13, 2022 at 4:06
  • $\begingroup$ What I mean is that $2 - 2\frac{<f_1(x),f_2(x)>}{||f_1(x)||_2 ||f_2(x)||_2}$ is valid if and only of norms equals 1, but in this sense also $2||f_2(x)||_2 -2\frac{<f_1(x),f_2(x)>}{||f_1(x)||_2}$ is also valid, and so on. The honest variant is simply $2 - 2<f_1(x),f_2(x)>$, it replaces everywhere the norm with $1$. $\endgroup$
    – rapaio
    Feb 13, 2022 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.