confused about parameter updates and forward/backward pass according to batches and epochs in CNN?

Question

I am working on a CNN model, the code written in tensorflow, I did some googling about parameter updates such as weights ana biases when method is optimized and the loss is computed, two things made me confuse:

1- after output layer, the data goes to loss, the loss compute and then the model is begin optimization or in reverse of that?

2- Is parameters updated after each mini-batch fed to network (i.e. the forward and backward pass is done for every batch) or only updates when one epoch is completed? why some tutorials said that each epoch is ba forward/backward pass?

anyone can clarify it please? if with a reference its better for me.

Answer 1

I understand your confusion, and the real cause of all this mismatch between different tutorials is because there is many equivalent ways to train a neural network when dealing with batches, epochs. However, I think it is best to stick with the most common terminology, i.e. that used by the deep learning libraries.

Backpropagation

For question 1, you are correct. We feed the data through the inputs, the data goes through a forward pass and then we obtain an output. With the output we can calculate a loss. Then we will use backpropagation to attribute some fault to each model parameter for the resulting error (loss). Then we will use gradient descent to update the model parameters accordingly. You can see how this works here.

Epochs and batches

For question 2. First let's define some terms. One epoch ends when all the training data available has been consumed. The second epoch goes through all the data again. In a simple neural network with not much data, you will pass all the training instances through the network successively and get the loss for each output. Then we will get an average of these losses to estimate the total loss for all instances. This results in one backpropagation per epoch.

However, most of the time it is not possible to fit all the data into memory so we must use batches, this means we will only feed-forward some training instances at a time. Then we will calculate the loss resulting from these instances and tune the parameters using backpropagation. Say we have 1000 training instances, then we can use a batch size of 100, we will thus do back-propagation 10 times per epoch.

Pros and cons

The pros of using batches is you can use larger datasets to train your model, however the smaller the batch size the less accurate the loss function estimate.