0
$\begingroup$

I am new to regressions and we are doing a very simple exercise in a course I am taking to get a basic understanding of GD and SGD for a linear regression.

From my understanding, the only difference between GD and SGD is that instead of performing the algorithm on dataset size m as is done in GD, SGD performs the operation on subsets of m.

My question is -- for SGD does one simply perform the algorithm on the mini-batch, or is there some sort of summation of the results to come out with a final answer? Apologies if I am not asking in the correct terms, I am newer to some of the mathematical concepts involved.

|improve this question
$\endgroup$
1
$\begingroup$

In SGD you just feed an example to your model, compute the gradient of the loss function of that example and update the weights according to the gradient of the loss of that example.

In mini-batch gradient descent you feed a batch to your model, compute the gradient of the loss of that batch and update the weights according to the gradient of the loss of that batch.

In fact, SGD is mini-batch gradient descent with batch size equal to 1.

|improve this answer
$\endgroup$
  • $\begingroup$ This was exactly what I was trying to figure out. Great, succinct answer. $\endgroup$ – foobarbaz Apr 24 '18 at 21:52
  • $\begingroup$ I need 2 more rep points before I can :) $\endgroup$ – foobarbaz Apr 25 '18 at 11:40
  • $\begingroup$ I tried to implement this, and included it in another post, I would be interested to see if you had thoughts on how I implemented the cost function and weight updates datascience.stackexchange.com/questions/30786/… $\endgroup$ – foobarbaz Apr 25 '18 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.