Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
My current understanding is that in SGD, after each data sample, the loss is used to update each weight. Ex: With 1000 samples and a network with 10 weights, there will be 10,000 individual weight updates per epoch.
In Gradient Descent and Batch Gradient Descent, how are these updates deferred over multiple data samples? What is being stored at each sample, that can be applied at the end of the batch? Is the loss at each sample averaged over the batch?
What is being stored at each sample, that can be applied at the end of the batch
You store the prediction and error and then calculate the average after each cycle.
GD - Update after all rows. It will give a correct direction to the weight update but will be very slow
SGD - Update after each row. It will be fast but the direction may swing
Batch - Update after a specific count. This approach gives a balanced solution to the two other approaches.
