# With Stochastic Gradient Descent why we dont compute exact derivative of loss function?

In a blog I read this:
With Stochastic Gradient Descent we don’t compute the exact derivate of our loss function. Instead, we’re estimating it on a small batch.
blog.
Now I am confused with the whole concept.
Why we take estimate of the derivative? Please explain.

• In context I presume they mean that we take an estimation of the derivative via minibatch as oppose to calculating the exact gradient as in "gradient descent" – chappers Sep 13 at 11:45
• How is there an estimation in mini batch? We calculate the loss for a small number of data points(batch size) till every data point is considered – Shiv Sep 14 at 5:14

That's because the whole loss is $$\frac{1}{N} \sum\limits_{i=1}^N L(x_i, y_i)$$ and that number $$N$$ is the dataset size, it can be very large. It's just too slow to compute the true gradient, thus we compute its unbiased estimate via Monte Carlo. There are some theorems that say that stochastic gradient descent converges under certain conditions, so it's a reasonable method. You just don't to wait long (computing the true gradient), you can converge faster. The speed isn't the only reason. Also, researchers found out that using small batch size can improve the performance of neural networks and it's reasonable as well because the lower the batch size the higher is the variance of the estimate, and the higher variance (i.e. noise) and the higher variance prevents the net from overfitting.
• We approximate $\frac{1}{N} \sum\limits_{i=1}^N L(x_i, y_i)$ with $\frac{1}{M} \sum\limits_{i=1}^M L(x_{i_j}, y_{i_j})$, where $M < N$ and $(x_{i_j}, y_{i_j})$ are randomly sampled from you dataset, this is our random minibatch. We update the weights with gradients of the loss approximation. Then sample another minibatch and do the same and repeat all until convergence (to actually converge we slowly decrease the learning rate). – Michael Solotky Sep 15 at 7:32