2
$\begingroup$

In batch gradient descent, it is said that one iteration of gradient descent update takes the processing of whole entire dataset, which I believe makes an epoch.On the other hand, in mini batch algorithm an update is made after every mini batch and once every mini batch is done, one epoch is completed. So in both cases, an epoch is completed after all the data is processed.I do not quite get what makes mini batch algorithm more efficient.

Thanks,

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

In short, batch gradient descent is accurate but plays it safe, and therefore is slow. Mini-batch gradient descent is a bit less accurate, but doesn't play it safe and is much faster.

When you do gradient descent, you use an estimate of the gradient to update your weights. When you use batch gradient descent, your gradient estimate is 100% accurate since it uses all your data.

Mini-batch is considered more efficient because you might be able to get, let's say, an ~80% accurate gradient with only 5% of the data (these numbers are made up). So, your weights may not always be updated optimally (if your estimate is not so good), but you will be able to update your weights more often since you don't need to go through all your data at once.

The idea is that you update your weights more often with an approximation of your gradient, which often is good enough. The utility of mini-batch becomes more obvious when you start dealing with very large datasets.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thank you for the great explanation but I guess I need to reiterate my question. To my understanding, let's say I am training my model for 200 epochs, in both batch GD and mini-batch GD, an epoch is completed once all the training data is processed.The only difference is, in mini batch the gradient is updated at the end of each mini batch whereas in batch GD, the gradient is updated at the end of the epoch only once all the data is processed..At the end, it seems both of the algorithms take similar amount of time to complete one epoch and I do not see the point of using mini batch. $\endgroup$
    – Arwen
    May 6, 2020 at 8:56
  • $\begingroup$ The point of using mini-batch is that you are able to update your weights more than once each epoch, so your model gets better. A mini-batch update should improve the model less than a batch update, but because you can do more updates, your model should be better using mini-batch $\endgroup$
    – Valentin Calomme
    May 6, 2020 at 9:03
  • 1
    $\begingroup$ So, I see that mini batch algorithm can sometimes be computationally more inefficient than batch GD as we are doing more frequent updates but it has the potential to save us from a local minima that a batch GD algorithm can be stuck onto. Thanks! $\endgroup$
    – Arwen
    May 6, 2020 at 9:31
  • $\begingroup$ Indeed, mini-batch adds some level of randomness since the gradient is estimated over a small batch of data, which can also help not being stuck in local minima. Most of the time, mini-batch is required, because if your dataset contains millions of training examples, you might only be able to update your network every few hours using batch gradient descent, which just isn't often enough $\endgroup$
    – Valentin Calomme
    May 6, 2020 at 9:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.