4
$\begingroup$

My current understanding is that in SGD, after each data sample, the loss is used to update each weight. Ex: With 1000 samples and a network with 10 weights, there will be 10,000 individual weight updates per epoch.

In Gradient Descent and Batch Gradient Descent, how are these updates deferred over multiple data samples? What is being stored at each sample, that can be applied at the end of the batch? Is the loss at each sample averaged over the batch?

$\endgroup$
1
  • $\begingroup$ I think the answer is, that they are not deferred at all in SGD but applied after each sample. And in batch gradient descent it is done after each batch. I found some interesting page about this here: geeksforgeeks.org/ml-stochastic-gradient-descent-sgd. The page states, that SGD is the same as batch gradient descent with batch-size 1 btw. $\endgroup$
    – jottbe
    Sep 27, 2019 at 2:17

1 Answer 1

1
$\begingroup$

What is being stored at each sample, that can be applied at the end of the batch

You store the prediction and error and then calculate the average after each cycle.

GD - Update after all rows. It will give a correct direction to the weight update but will be very slow

SGD - Update after each row. It will be fast but the direction may swing

Batch - Update after a specific count. This approach gives a balanced solution to the two other approaches.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.