Is training one epoch using mini-batch gradient descent slower than using batch gradient descent?

Question

I wonder whether one epoch using mini-batch gradient descent is slower than one epoch using just batch gradient descent.

At least I understand that one iteration of mini-batch gradient descent should be faster than one iteration of batch gradient descent.

However, if I understand it correctly, since the mini-batch gradient descent must update the weights by the number of the batch size in one epoch, the training would be slower than the batch gradient descent, which computes and updates the weights only once in one epoch.

Is this correct? In that case, is it worth worrying about the loss of the overall training time?

Answer 1

You are correct, there is more overhead to process the same amount of data because you do more weight updates and maybe preprocessing the batches in your generator will take more time as well. However, since you are doing more updates and if your batch size is big enough, the gradients of your mini-batches will approximate the gradients of your full batch fairly well and you will have gotten much closer to the optimum of your full dataset by splitting your full batch into mini batches than you would have gotten by doing one non-stochastic weight update.