0
$\begingroup$

I'm new to regression and we are doing a very simple exercise in a course. I'm taking to get a basic understanding of GD and SGD for a linear regression.

From my understanding, the only difference between GD and SGD is that instead of performing the algorithm on dataset size m as is processing in GD, SGD performs the operation on subsets of m.

My question is, for SGD does one simply perform the algorithm on the mini-batch, or is there some sort of summation of the results to come out with a final answer? Apologies if I'm not asking in the correct terms, I'm newer to some of the mathematical concepts involved.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In SGD you just feed an example to your model, compute the gradient of the loss function of that example and update the weights according to the gradient of the loss of that example.

In mini-batch gradient descent you feed a batch to your model, compute the gradient of the loss of that batch and update the weights according to the gradient of the loss of that batch.

In fact, SGD is mini-batch gradient descent with batch size equal to 1.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This was exactly what I was trying to figure out. Great, succinct answer. $\endgroup$
    – foobarbaz
    Apr 24, 2018 at 21:52
  • $\begingroup$ I need 2 more rep points before I can :) $\endgroup$
    – foobarbaz
    Apr 25, 2018 at 11:40
  • $\begingroup$ I tried to implement this, and included it in another post, I would be interested to see if you had thoughts on how I implemented the cost function and weight updates datascience.stackexchange.com/questions/30786/… $\endgroup$
    – foobarbaz
    Apr 25, 2018 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.