5
$\begingroup$

[I've cross-posted it to cross.validated because I'm not sure where it fits best]

How does gradient descent work for training a neural network if I choose mini-batch (i.e., sample a subset of the training set)? I have thought of three different possibilities:

Epoch starts. We sample and feedforward one minibatch only, get the error and backprop it, i.e. update the weights. Epoch over.

Epoch starts. We sample and feedforward a minibatch, get the error and backprop it, i.e. update the weights. We repeat this until we have sampled the full data set. Epoch over.

Epoch starts. We sample and feedforward a minibatch, get the error and store it. We repeat this until we have sampled the full data set. We somehow average the errors and backprop them by updating the weights. Epoch over.

Improve this question
$\endgroup$
4
  • $\begingroup$ Cross Posting is not encouraged in SE :) $\endgroup$
    – Dawny33
    Dec 14, 2015 at 17:49
  • $\begingroup$ I'm well aware of that. I wasn't sure where it fits best. $\endgroup$
    – Alex
    Dec 15, 2015 at 10:24
  • $\begingroup$ I think it is best suited here! $\endgroup$
    – Dawny33
    Dec 15, 2015 at 10:26
  • 1
    $\begingroup$ Cross-posted: datascience.stackexchange.com/q/9378/8560, stats.stackexchange.com/q/186687/2921. Please do not post the same question on multiple sites. Each community should have an honest shot at answering without anybody's time being wasted. If you're not sure where to post it, pick one place that seems best. If you don't get an answer after a week or so, you can delete it and post it elsewhere, or flag for moderator attention and ask the moderators to migrate it. $\endgroup$
    – D.W.
    Jan 4, 2017 at 22:43

2 Answers 2

Reset to default
7
$\begingroup$

Let us say that the output of one neural network given it's parameters is $$f(x;w)$$ Let us define the loss function as the squared L2 loss (in this case). $$L(X,y;w) = \frac{1}{2n}\sum_{i=0}^{n}[f(X_i;w)-y_i]^2$$ In this case the batchsize will be denoted as $n$. Essentially what this means is that we iterate over a finite subset of samples with the size of the subset being equal to your batch-size, and use the gradient normalized under this batch. We do this until we have exhausted every data-point in the dataset. Then the epoch is over. The gradient in this case: $$\frac{\partial L(X,y;w)}{\partial w} = \frac{1}{n}\sum_{i=0}^{n}[f(X_i;w)-y_i]\frac{\partial f(X_i;w)}{\partial w}$$ Using batch gradient descent normalizes your gradient, so the updates are not as sporadic as if you have used stochastic gradient descent.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks, so if $n$ is the size of the mini-batch, we update the weights after each mini-batch, right: $\frac{\partial L}{ \partial w}$, so if we have 100 mini-batches, we update weights 100 times in 1 epoch. If I understood you correctly, I'm not sure where normalized gradient (i.e. divided by the norm) fits. $\endgroup$
    – Alex
    Dec 15, 2015 at 21:59
  • 1
    $\begingroup$ @Alex Yep exactly. 100 mini-batches means updating the weights 100 times. For the normalizing the gradient I was just explaining why mini-batch is preferred to stochastic gradient descent. $\endgroup$
    – Armen Aghajanyan
    Dec 15, 2015 at 22:16
2
$\begingroup$

When you train with mini-batches then you have the second option, network is updated after each mini-batch, and epoch ends after presenting all samples.

Please see these responses

Improve this answer
$\endgroup$
1
  • $\begingroup$ Ok. Can you point me to a specific publication detailing it? The most i've seen skim over it $\endgroup$
    – Alex
    Dec 14, 2015 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.