I understand pretty OK how to derive the formulas and implement stochastic gradient descent for a deep neural network (even though the total derivative magic for hidden layers is a bit pushing my limits).

I'm struggling to grasp the fundamentals of generalizing the method to batch or mini batch training.

From what I understand, in batch training you feed forward all of your batch's examples and then backpropagate the error.

In stochastic backpropagation most computations involve taking the derivative of the error with respect to neurons' activations. But in the case of batch training there is one activation per training example. Do you average them or something ? Or is the math radically different?

Is there a simple example somewhere that doesn't involve tensors?


Do you average them or something ?

Yes! The gradient of the loss for the minibatch is just a simple average of the gradients of the individual examples in the minibatch. See Section 8.1.3 of the Deep Learning Book:

Hence, we can obtain an unbiased estimator of the exact gradient of the generalization error by sampling a minibatch of examples $\{x(1),...x(m)\}$ with corresponding targets $y(i)$ from the data-generating distribution $p_{data}$, then computing the gradient of the loss with respect to the parameters for that minibatch: $$\hat{g} = \frac{1}{m} ∇_\theta \sum_{i} L(f(x^{(i)}; \theta), y^{(i)}).$$

| improve this answer | |
  • $\begingroup$ Thanks, so whenever I see a "a" or "g'(a)" where a is a neuron's activation value and g an activation function in my formulas I just replace them with their averages over the batch size? Or am I oversimplifying? $\endgroup$ – djfm Aug 22 '18 at 17:54
  • $\begingroup$ For instance does the recursive formula for getting the error signal delta as seen here still kind of hold? medium.com/@erikhallstrm/… $\endgroup$ – djfm Aug 22 '18 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.