I've recently been writing linear regression algorithms from scratch to gain an understanding of how the maths behind it works (something that was a bit of a black box beforehand), and so I got around to differentiating the cost function. Without realising it I used the Squared Error for the cost function - the MSE but without dividing by the dataset length. Is there any benefit (faster approach of the minimum or other) to using the Mean Squared Error over just summing the squares of the error?
As the gradients are calculated from the loss, it is different. Depending on the batch size the learning rate should be lowered when using
tf.reduce_sum or other summary method. Both can yield a successful training, however there is one catch.
Batch sizes sometimes may vary because the last batch is smaller or maybe you vary your batch size just for fun (provided you have built your graph with this possibility). In this case
reduce_sum will cause fluctuating loss values instead of just decreasing. You want to avoid this.
reduce_sum and use
reduce_mean for linear regression, although with a higher learning rate.
Just for reference a four parameter Linear regression:
$$y = b_0 + b_1*x_1 + b_2*x_2 + b_3*x_3$$
batch size of 100 and a 0.01 learning rate for GradientDescent yields a MSE of 16.93 in 100 epochs
batch size of 100 and a 0.0001 learning rate for GradientDescent yields a SSE of 1693.31 in 100 epochs
which is exactly the same, and also yields the same parameters (bias & weights), with random-seeds locked.