$$ C(w,b) = \frac{1}{2n}\sum_{x}||y(x)-a||^2 $$
Where y is a 10-dimensional vector, a is the output, w is the weight and b is the bias and n is the number of inputs. If this is the MSE, shouldn't it be $\frac{1}{n}$ instead?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
This is really just like a convention that appears in some places because we normallt want to take the derivative of the cost function (i.e. compute gradients), which means the power of 2 would be taken to the front.
If we put the $\frac{1}{2}$ at the front to begin with, it just looks nicer once we have finished. I have seen this written somewhere before in a paper, but can't find a reference right now.
Because the nominal values of cost themselves (the scale of the values) is not of importance, we can scale it as we like really. Multiplying by a constance of $0.5$ does not change the algebraic behaviour.
-
$\begingroup$ Thanks. That does make sense. Could you please leave the link if you happen to remember it? $\endgroup$ Jun 13, 2018 at 15:12
-
1$\begingroup$ @AniketChowdhury - I will add any links I can find - or maybe someone else can add links as a comment :) $\endgroup$– n1k31t4Jun 13, 2018 at 16:41
$\begingroup$
$\endgroup$
1
It's pretty common to formulate quadratic loss in that way, because $\frac{d}{dx}\frac{(x-x')^2}{2} = x - x'$.
If you're only interested in minimizing that quantity then it doesn't matter if you rescale it by a constant, since locations of extrema would be the same.
-
$\begingroup$ Yeah. Thanks. I should have figured that out. Sorry for the bother. $\endgroup$ Jun 13, 2018 at 15:13