How do GD, Batch GD, SGD, and Mini-Batch SGD differ?

Question

How do these four types of gradient descent functions differ from each other?

• GD
• Batch GD
• SGD
• Mini-Batch SGD

Answer 1

Gradient Descent is an optimization method used to optimize the parameters of a model using the gradient of an objective function ( loss function in NN ). It optimizes the parameters until the value of the loss function is the minimum ( of we've reached the minima of the loss function ). It is often referred to as back propagation in terms of Neural Networks.

All the below methods are variants of Gradient Descent. You can learn more from this video.

Batch Gradient Descent:

The samples from the whole dataset are used to optimize the parameters i.e to compute the gradients for a single update. For a dataset of 100 samples, updates occur only once.

Stochastic Gradient Descent:

Stochastic GD computes the gradients for each and every sample in the dataset and hence makes an update for every sample in the dataset. For a dataset of 100 samples, updates occur 100 times.

Mini Batch Gradient Descent:

This is meant to capture the good aspects of Batch and Stochastic GD. Instead of a single sample ( Stochastic GD ) or the whole dataset ( Batch GD ), we take small batches or chunks of the dataset and update the parameters accordingly. For a dataset of 100 samples, if the batch size is 5 meaning we have 20 batches. Hence, updates occur 20 times.

All the above methods use gradient descent for optimization. The main difference is that on how much samples are the gradients calculated. Gradients are averaged in Mini-Batch and Batch GD.

You can refer to these blogs/posts:

Batch gradient descent versus stochastic gradient descent

Gradient Descent Algorithm and Its Variants

Answer 2

Though GD's has been explained very well, I would like to add few more points on it through a hypothetical situation.

Consider we have a data set of very large values say in Millions.

1. GD: This will update the weights only after calculating the mean loss of all the samples. Hence in such situation it will become very costly operation and it will converge very slowly.
2. Batch GD/Mini-Batch GD: Alternative of GD. Selects a batch size and overcome the above said problem of GD. But still executes in batch.
3. SGD: Update weights with every sample and converge very fast.

So one can argue, why we should not always use SGD?

Since SGD update weights with every sample, its results are not that precise and hence it will not help us to converge to the best solution or we can also say that not allow us to get the Best Global Minima, rather it moves close to Global Minimum value.

Note: I suggest to view this video on SGD. Atleast it cleared most of my doubt.

Answer 3

Gradient descent (GD) refers to the general optimisation method that uses the gradient of the loss function to update the values of the parameters of the model in the "direction" of the steepest descent. GD can thus refer to batch GD, SGD or mini-batch SGD.

SGD refers to GD that updates the parameters of your model after every single labelled pair $(\boldsymbol{x}, y)$, where $\boldsymbol{x}$ is an observation and $y$ is the corresponding label or class (in the case of classification tasks).

Batch GD and mini-batch SGD are (usually) synonous, and they refer to a version of the GD method where the parameters are updated using one or more labelled pairs (denoted by "batch" or "mini-batch"). See this for more details.

However, note that, in general, some people might not use these terms according to their definitions above. For example, some people might use SGD to refer to mini-batch SGD.