4
$\begingroup$

In the famous Deep Learning Book, in chapter 1, equation 6, the Quadratic Cost (or Mean Squared Error) in a neural network is defined as

$ C(w, b) = \frac{1}{2n}\sum_{x}||y(x)-a||^2 $

where $w$ is the set of all weights and $b$ the set of all biases, $n$ is the number of training inputs, x is the set of all training inputs, y(x) is the expected output of the network for input x, and $a$ is the actual output of the network for input $x$, with respect to $w$ and $b$.

Most of this formula seems very clear to be, except the $2$ in the denominator. If I understand it correctly, we are summing up the squared vector length of (the actual output minus its expected output), for each training input (giving us the total squared error for the training set) and then divide this by the number of training samples, to get the mean squared error of all training samples. Why do we divide this by $2$ as well then?

In other places I've seen that Andrew Ng's lecture defines the Mean Square cost in a similar way, also with the $2$ in the denominator, so this seems to be a common definition.

$\endgroup$
4
$\begingroup$

This is just for mathematical convenience. When you differentiate $C(w,b)$, you will get an extra $2$. To eliminate that, $2$ is kept beforehand in denominator.

You can also watch this video on SVM lecture by Patrick Winston, where he uses a similar formula and then tells that he is using $2$ in denominator just for mathematical convenience.

$\endgroup$
0
$\begingroup$

It's just that it helps when we do the backprop of error while differentiating, as the 2 in the denominator cancels out with the 2 which we will get from the exponent..

So it's just a matter of Mathematical Convenience...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.