Gradient descent vs stochastic gradient descent vs mini-batch gradient descent with respect to working step/example

Question

I am trying to understand the working of gradient descent, stochastic gradient descent and mini-batch gradient descent.

In case of gradient descent, gradient is computed on the entire dataset at each step. So I imagine this is like multiple tasks, where-in each task looks at 1 item from the dataset. Eventually the result of the best task is selected.

Where-as stochastic gradient descent gets a new random sample at each step. So unlike the multiple tasks example above, there is only 1 task, and at each step of the task a random sample is selected. I assume the new random sample that is selected must be at a lower location then the existing sample?

In case of mini-batch gradient descent, I imagine this like a single main task, where-in each step results in multiple tasks, where-in each task will perform the gradient loss calculation and an average will be taken. Then the same process will repeat in the next step.

Answer 1

Your question is closely related to the way neural networks learn. the learning of a neural network is iterative. At each iteration, there is a forward pass and a backward pass

Forward pass : you select a batch of your observations and you send it to the neural networks. By going forward, your neural network will output predictions and compare it with the label of the observations(here the animal on the picture). The neural network will then compute the cost function, meaning the average error made by the model on the observations you have passed.

Backward pass : you will update the different parameters starting from the end and coming back to the first parameters. Weights will by updated with respect to the error you have made in average. Once the backward pass is done, you can start a new iteration, meaning a forward pass + backward pass.

Difference between full batch and mini-batch learning ?

Full batch learning : as you said, all the observations of the dataset are used in the batch. However, there are processed at the same time. There is a unique forward pass and backward pass. It is achieved using vectorization. The cost function is computed as an average of the errors made on all the observations.

Mini-batch learning : you will only use a subset of your observations (let's say "K") at the same time. Once you have runned your forward and backward passes, you will pick an other subset of the same size (with no replacement possible) and make a new iterations. Once your model has seen all the observations of your batch, you have achieved a "epoch". You can then start a new epoch by shuffling your observations and create new batchs. It is the also called stochastic gradient descent.

How to choose the batch size ?

It will depend of the size of your dataset. With a large batch size, the gradient descent is robust but the forward-backward pass is slower to compute. With a small batch size, the gradient descent will be less robust but each iteration is faster to compute.