0
$\begingroup$

In a blog I read this:
With Stochastic Gradient Descent we don’t compute the exact derivate of our loss function. Instead, we’re estimating it on a small batch.
blog.
Now I am confused with the whole concept.
Why we take estimate of the derivative? Please explain.

$\endgroup$
2
  • $\begingroup$ In context I presume they mean that we take an estimation of the derivative via minibatch as oppose to calculating the exact gradient as in "gradient descent" $\endgroup$
    – chappers
    Sep 13, 2020 at 11:45
  • $\begingroup$ How is there an estimation in mini batch? We calculate the loss for a small number of data points(batch size) till every data point is considered $\endgroup$
    – Shiv
    Sep 14, 2020 at 5:14

1 Answer 1

3
$\begingroup$

That's because the whole loss is $\frac{1}{N} \sum\limits_{i=1}^N L(x_i, y_i)$ and that number $N$ is the dataset size, it can be very large. It's just too slow to compute the true gradient, thus we compute its unbiased estimate via Monte Carlo. There are some theorems that say that stochastic gradient descent converges under certain conditions, so it's a reasonable method. You just don't to wait long (computing the true gradient), you can converge faster. The speed isn't the only reason. Also, researchers found out that using small batch size can improve the performance of neural networks and it's reasonable as well because the lower the batch size the higher is the variance of the estimate, and the higher variance (i.e. noise) and the higher variance prevents the net from overfitting.

$\endgroup$
5
  • $\begingroup$ So it means , that in normal gradient descent we calculate exact derivative as opposed to sgd where we estimate?? $\endgroup$
    – Shiv
    Sep 14, 2020 at 5:00
  • $\begingroup$ Yep, that's right $\endgroup$ Sep 14, 2020 at 8:14
  • $\begingroup$ How is there an estimation in mini batch??We calculate loss for a batch ,update weights and then calculate loss for every other example. $\endgroup$
    – Shiv
    Sep 15, 2020 at 5:29
  • 1
    $\begingroup$ We approximate $\frac{1}{N} \sum\limits_{i=1}^N L(x_i, y_i)$ with $\frac{1}{M} \sum\limits_{i=1}^M L(x_{i_j}, y_{i_j})$, where $M < N$ and $(x_{i_j}, y_{i_j})$ are randomly sampled from you dataset, this is our random minibatch. We update the weights with gradients of the loss approximation. Then sample another minibatch and do the same and repeat all until convergence (to actually converge we slowly decrease the learning rate). $\endgroup$ Sep 15, 2020 at 7:32
  • $\begingroup$ Thank you so much.. $\endgroup$
    – Shiv
    Sep 15, 2020 at 7:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.