In a blog I read this:
With Stochastic Gradient Descent we don’t compute the exact derivate of our loss function. Instead, we’re estimating it on a small batch.
Now I am confused with the whole concept.
Why we take estimate of the derivative? Please explain.

  • $\begingroup$ In context I presume they mean that we take an estimation of the derivative via minibatch as oppose to calculating the exact gradient as in "gradient descent" $\endgroup$ – chappers Sep 13 '20 at 11:45
  • $\begingroup$ How is there an estimation in mini batch? We calculate the loss for a small number of data points(batch size) till every data point is considered $\endgroup$ – Shiv Sep 14 '20 at 5:14

That's because the whole loss is $\frac{1}{N} \sum\limits_{i=1}^N L(x_i, y_i)$ and that number $N$ is the dataset size, it can be very large. It's just too slow to compute the true gradient, thus we compute its unbiased estimate via Monte Carlo. There are some theorems that say that stochastic gradient descent converges under certain conditions, so it's a reasonable method. You just don't to wait long (computing the true gradient), you can converge faster. The speed isn't the only reason. Also, researchers found out that using small batch size can improve the performance of neural networks and it's reasonable as well because the lower the batch size the higher is the variance of the estimate, and the higher variance (i.e. noise) and the higher variance prevents the net from overfitting.

  • $\begingroup$ So it means , that in normal gradient descent we calculate exact derivative as opposed to sgd where we estimate?? $\endgroup$ – Shiv Sep 14 '20 at 5:00
  • $\begingroup$ Yep, that's right $\endgroup$ – Michael Solotky Sep 14 '20 at 8:14
  • $\begingroup$ How is there an estimation in mini batch??We calculate loss for a batch ,update weights and then calculate loss for every other example. $\endgroup$ – Shiv Sep 15 '20 at 5:29
  • 1
    $\begingroup$ We approximate $\frac{1}{N} \sum\limits_{i=1}^N L(x_i, y_i)$ with $\frac{1}{M} \sum\limits_{i=1}^M L(x_{i_j}, y_{i_j})$, where $M < N$ and $(x_{i_j}, y_{i_j})$ are randomly sampled from you dataset, this is our random minibatch. We update the weights with gradients of the loss approximation. Then sample another minibatch and do the same and repeat all until convergence (to actually converge we slowly decrease the learning rate). $\endgroup$ – Michael Solotky Sep 15 '20 at 7:32
  • $\begingroup$ Thank you so much.. $\endgroup$ – Shiv Sep 15 '20 at 7:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.