5
$\begingroup$

[I've cross-posted it to cross.validated because I'm not sure where it fits best]

How does gradient descent work for training a neural network if I choose mini-batch (i.e., sample a subset of the training set)? I have thought of three different possibilities:

Epoch starts. We sample and feedforward one minibatch only, get the error and backprop it, i.e. update the weights. Epoch over.

Epoch starts. We sample and feedforward a minibatch, get the error and backprop it, i.e. update the weights. We repeat this until we have sampled the full data set. Epoch over.

Epoch starts. We sample and feedforward a minibatch, get the error and store it. We repeat this until we have sampled the full data set. We somehow average the errors and backprop them by updating the weights. Epoch over.

$\endgroup$
  • $\begingroup$ Cross Posting is not encouraged in SE :) $\endgroup$ – Dawny33 Dec 14 '15 at 17:49
  • $\begingroup$ I'm well aware of that. I wasn't sure where it fits best. $\endgroup$ – Alex Dec 15 '15 at 10:24
  • $\begingroup$ I think it is best suited here! $\endgroup$ – Dawny33 Dec 15 '15 at 10:26
  • 1
    $\begingroup$ Cross-posted: datascience.stackexchange.com/q/9378/8560, stats.stackexchange.com/q/186687/2921. Please do not post the same question on multiple sites. Each community should have an honest shot at answering without anybody's time being wasted. If you're not sure where to post it, pick one place that seems best. If you don't get an answer after a week or so, you can delete it and post it elsewhere, or flag for moderator attention and ask the moderators to migrate it. $\endgroup$ – D.W. Jan 4 '17 at 22:43
7
$\begingroup$

Let us say that the output of one neural network given it's parameters is $$f(x;w)$$ Let us define the loss function as the squared L2 loss (in this case). $$L(X,y;w) = \frac{1}{2n}\sum_{i=0}^{n}[f(X_i;w)-y_i]^2$$ In this case the batchsize will be denoted as $n$. Essentially what this means is that we iterate over a finite subset of samples with the size of the subset being equal to your batch-size, and use the gradient normalized under this batch. We do this until we have exhausted every data-point in the dataset. Then the epoch is over. The gradient in this case: $$\frac{\partial L(X,y;w)}{\partial w} = \frac{1}{n}\sum_{i=0}^{n}[f(X_i;w)-y_i]\frac{\partial f(X_i;w)}{\partial w}$$ Using batch gradient descent normalizes your gradient, so the updates are not as sporadic as if you have used stochastic gradient descent.

$\endgroup$
  • 1
    $\begingroup$ Thanks, so if $n$ is the size of the mini-batch, we update the weights after each mini-batch, right: $\frac{\partial L}{ \partial w}$, so if we have 100 mini-batches, we update weights 100 times in 1 epoch. If I understood you correctly, I'm not sure where normalized gradient (i.e. divided by the norm) fits. $\endgroup$ – Alex Dec 15 '15 at 21:59
  • 1
    $\begingroup$ @Alex Yep exactly. 100 mini-batches means updating the weights 100 times. For the normalizing the gradient I was just explaining why mini-batch is preferred to stochastic gradient descent. $\endgroup$ – Armen Aghajanyan Dec 15 '15 at 22:16
2
$\begingroup$

When you train with mini-batches then you have the second option, network is updated after each mini-batch, and epoch ends after presenting all samples.

Please see these responses

$\endgroup$
  • $\begingroup$ Ok. Can you point me to a specific publication detailing it? The most i've seen skim over it $\endgroup$ – Alex Dec 14 '15 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.