Skip to main content

Questions tagged [mini-batch-gradient-descent]

Is a variation of the gradient descent algorithm that splits the training dataset into small batches that are used to calculate model error and update model coefficients. Implementations may choose to sum the gradient over the mini-batch which further reduces the variance of the gradient. The point of using mini-batch is that you are able to update your weights more than once each epoch, so your model gets better. Mini-batch is considered more efficient.

Filter by
Sorted by
Tagged with
23 votes
2 answers
18k views

Sliding window leads to overfitting in LSTM?

Will I overfit my LSTM if I train it via the sliding-window approach? Why do people not seem to use it for LSTMs? For a simplified example, assume that we have to predict the sequence of characters: <...
Kari's user avatar
  • 2,736
8 votes
2 answers
1k views

Why is taking the gradient of the average error in SGD not correct, but rather the average of the gradients of single errors?

I am a little confused about taking averages in cost functions and SGD. So far I always thought in SGD you would compute the average error for a batch and then backpropagate it. But then I was told in ...
lo tolmencre's user avatar
14 votes
2 answers
4k views

Why averaging the gradient works in Gradient Descent?

In Full-batch Gradient descent or Minibatch-GD we are getting gradient from several training examples. We then average them out to get a "high-quality" gradient, from several estimations and finally ...
Kari's user avatar
  • 2,736