How do you generalize from stochastic to batch learning using gradient descent?

Question

I understand pretty OK how to derive the formulas and implement stochastic gradient descent for a deep neural network (even though the total derivative magic for hidden layers is a bit pushing my limits).

I'm struggling to grasp the fundamentals of generalizing the method to batch or mini batch training.

From what I understand, in batch training you feed forward all of your batch's examples and then backpropagate the error.

In stochastic backpropagation most computations involve taking the derivative of the error with respect to neurons' activations. But in the case of batch training there is one activation per training example. Do you average them or something ? Or is the math radically different?

Is there a simple example somewhere that doesn't involve tensors?

Answer 1

Do you average them or something ?

Yes! The gradient of the loss for the minibatch is just a simple average of the gradients of the individual examples in the minibatch. See Section 8.1.3 of the Deep Learning Book:

Hence, we can obtain an unbiased estimator of the exact gradient of the generalization error by sampling a minibatch of examples $\{x(1),...x(m)\}$ with corresponding targets $y(i)$ from the data-generating distribution $p_{data}$, then computing the gradient of the loss with respect to the parameters for that minibatch: $$\hat{g} = \frac{1}{m} ∇_\theta \sum_{i} L(f(x^{(i)}; \theta), y^{(i)}).$$