How many times is backprop used in epoch?

Question

As I understand for the algorithms that use gradient descent we have to pass data to the algorithms multiple times so that the optimum is found.

So one epoch means that the forward-backprop (and updating weights with gradient descent) is done only once and in order to find the minimum we need to run the algorithm more times?

Or is forward-backprop combination done multiple times (like for every input of data that is passed to the input layer) in one epoch until the minimum for that epoch is found?

Answer 1

It depends on the type of gradient descent or respectively your batch size: One epoch means that your neural net (NN) has applied the forward pass on all examples of your training data, i.e. it has "seen" all training data. Now to do so you have at least two options (let $n$ be the number of samples in your training data):

1. You can either run backprop after each forward pass of an example and update your weights accordingly. This is usually implemented as stochastic gradient descent (SGD), has a batch size of 1 and means you run backprop $n$ times per epoch.

2. Alternatively, you can first run the forward pass for all your training data and then run backprop and update your weights accordingly. This is called batch gradient descent, has a batch size of $n$ and means you run backprop once per epoch.

Usually a mix of these two is applied so that $1 < \text{batch size} < n$, which is called Minibatch Gradient Descent. Following the above explanations, it means you'll run backprop $\frac{n}{\text{batch size}}$ times per epoch.

This blog post provides a good explanation as well.