I've recently been writing linear regression algorithms from scratch to gain an understanding of how the maths behind it works (something that was a bit of a black box beforehand), and so I got around to differentiating the cost function. Without realising it I used the Squared Error for the cost function - the MSE but without dividing by the dataset length. Is there any benefit (faster approach of the minimum or other) to using the Mean Squared Error over just summing the squares of the error?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ You need to checkout different Gradient Descent techs like batch, stochastic , Then different type of algos like Nesterov, Adding Momentum etc.. $\endgroup$– AdityaJun 15, 2018 at 4:23
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
No, it is exactly the same. Optimizing a function and the same function divided by a constant is equivalent, both in the analytical and the numerical sense. You will get exactly the same optimal parameters.
answered Jun 15, 2018 at 7:06
-
$\begingroup$ Thanks for the answer. I finished the gradient descent algorithm, and I did find that if I used the non-mean SE, I needed to use a lower learning rate. I'm guessing that this is because the cost function was amplified, so the derivative was also amplified, and therefore was more likely to overstep? $\endgroup$ Jun 16, 2018 at 8:18
-
$\begingroup$ Exactly, if the cost function is the double then the step size should be the half. $\endgroup$ Jun 16, 2018 at 8:31
-
1$\begingroup$ I dont agree with @DavidMasip it will not be same when you are using different values of gradients.Actually our overall training is based on how we are calculating the gradients.So taking a mean gonna land us on different place of training and it has many other reasons why we are using mean here! $\endgroup$ Jun 23, 2018 at 7:44
$\begingroup$
$\endgroup$
As the gradients are calculated from the loss, it is different. Depending on the batch size the learning rate should be lowered when using tf.reduce_sum or other summary method. Both can yield a successful training, however there is one catch.
Batch sizes sometimes may vary because the last batch is smaller or maybe you vary your batch size just for fun (provided you have built your graph with this possibility). In this case reduce_sum will cause fluctuating loss values instead of just decreasing. You want to avoid this.
TLDR: avoid reduce_sum and use reduce_mean for linear regression, although with a higher learning rate.
Just for reference a four parameter Linear regression:
$$y = b_0 + b_1*x_1 + b_2*x_2 + b_3*x_3$$
batch size of 100 and a 0.01 learning rate for GradientDescent yields a MSE of 16.93 in 100 epochs
batch size of 100 and a 0.0001 learning rate for GradientDescent yields a SSE of 1693.31 in 100 epochs
which is exactly the same, and also yields the same parameters (bias & weights), with random-seeds locked.
