# Explanation behind the calculation of training loss in deep learning model

I am trying to model an image classification problem using convolution neural network. I came across a code on Github in which I am not able to understand the meaning of following line for loss calculation in the training loop.
I am omitting most of the detail and only placing the relevent code-

for batch_idx, (data, target) in enumerate(final_train_loader):
loss = criterion(output,target)
#Idea behind the below line
train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))


Cross-entropy loss function is being used here.

train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))


is basically calculating the average train_loss for the finished batches

To illustrate, suppose 4 batches have been done (with average loss named avg_loss) and current is calculated from 5th batch (with loss named new_loss)

The new average loss is from

$$\frac {4 \times \text{avg_loss} + \text{new_loss}} {5}$$

This is exactly the same as

$$\text{avg_loss} + \frac {\text{new_loss} - \text{avg_loss}} {5}$$

which is the calculation done by the code

• Thanks Wang. Could you please explain me in form of some equation or by giving an instance or if you could point to some resources. I am finding it hard to build an intuition of it as I also find the following train_loss in many cases-train_loss += loss.item()*data.size(0) – Mark Sep 23 '19 at 2:07
• No worries. I've updated with the equations. – 1tan Sep 23 '19 at 2:12
• Another noob question, why we multiplied avg_loss with 4?Edit:Oh.. I got it...thanks. – Mark Sep 23 '19 at 2:15
• When calculating averages, we need its total amount. Multiply avg_loss with 4 to get the total amount of loss in the first 4 batches because avg_loss of the first 4 batches is calculated from total amount divided by 4. – 1tan Sep 23 '19 at 2:18
• you are really superb – Mark Sep 23 '19 at 2:37