2
$\begingroup$

While coding the batch gradient descent, it is easy to code the convergence as after each iterations the cost moves towards minimum and when the change in cost tends to approach a pre-defined number, we stop the iterations and conclude our gradient descent has converged. But in stochastic GD, the cost tend to wander off the local mimina at one point of time. Deciding a threshold of change does'nt work here as SGD does not always move to converge, rather fluctuates a lot. While coding in python, how do I know the number of iterations when the cost tends to be at minimum?

$\endgroup$
2
  • $\begingroup$ if you are updating weights after each training sample. you can calculate the cost as the average cost of the training samples in each epoch in order to determine if the algorithm converged. $\endgroup$
    – cap
    Nov 27, 2019 at 20:37
  • $\begingroup$ With SGD, do you mean pure Stochastic GD (when you feed one observation at a time), or mini batch GD (when you feed a batch of data of size n at each iteration)? I asked because different sources use these two labels interchangeably, but their interpretations are very different. $\endgroup$
    – Leevo
    Jan 2, 2020 at 8:59

1 Answer 1

0
$\begingroup$

Given the slight fluctuations in stochastic gradient descent, either average over several recent runs or set an epsilon/error value for minimal improvement.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.