# With Stochastic Gradient Descent why we dont compute exact derivative of loss function?

In a blog I read this:
With Stochastic Gradient Descent we don’t compute the exact derivate of our loss function. Instead, we’re estimating it on a small batch.
blog.
Now I am confused with the whole concept.
Why we take estimate of the derivative? Please explain.

• In context I presume they mean that we take an estimation of the derivative via minibatch as oppose to calculating the exact gradient as in "gradient descent" Sep 13, 2020 at 11:45
• How is there an estimation in mini batch? We calculate the loss for a small number of data points(batch size) till every data point is considered
– Shiv
Sep 14, 2020 at 5:14

That's because the whole loss is $$\frac{1}{N} \sum\limits_{i=1}^N L(x_i, y_i)$$ and that number $$N$$ is the dataset size, it can be very large. It's just too slow to compute the true gradient, thus we compute its unbiased estimate via Monte Carlo. There are some theorems that say that stochastic gradient descent converges under certain conditions, so it's a reasonable method. You just don't to wait long (computing the true gradient), you can converge faster. The speed isn't the only reason. Also, researchers found out that using small batch size can improve the performance of neural networks and it's reasonable as well because the lower the batch size the higher is the variance of the estimate, and the higher variance (i.e. noise) and the higher variance prevents the net from overfitting.

• So it means , that in normal gradient descent we calculate exact derivative as opposed to sgd where we estimate??
– Shiv
Sep 14, 2020 at 5:00
• Yep, that's right Sep 14, 2020 at 8:14
• How is there an estimation in mini batch??We calculate loss for a batch ,update weights and then calculate loss for every other example.
– Shiv
Sep 15, 2020 at 5:29
• We approximate $\frac{1}{N} \sum\limits_{i=1}^N L(x_i, y_i)$ with $\frac{1}{M} \sum\limits_{i=1}^M L(x_{i_j}, y_{i_j})$, where $M < N$ and $(x_{i_j}, y_{i_j})$ are randomly sampled from you dataset, this is our random minibatch. We update the weights with gradients of the loss approximation. Then sample another minibatch and do the same and repeat all until convergence (to actually converge we slowly decrease the learning rate). Sep 15, 2020 at 7:32
• Thank you so much..
– Shiv
Sep 15, 2020 at 7:39