I am trying to model an image classification problem using convolution neural network. I came across a code on Github in which I am not able to understand the meaning of following line for loss calculation in the training loop.
I am omitting most of the detail and only placing the relevent code-

for batch_idx, (data, target) in enumerate(final_train_loader):
     loss = criterion(output,target)
     #Idea behind the below line
     train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))

Cross-entropy loss function is being used here.


The line you're asking about

train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))

is basically calculating the average train_loss for the finished batches

To illustrate, suppose 4 batches have been done (with average loss named avg_loss) and current is calculated from 5th batch (with loss named new_loss)

The new average loss is from

$\frac {4 \times \text{avg_loss} + \text{new_loss}} {5}$

This is exactly the same as

$\text{avg_loss} + \frac {\text{new_loss} - \text{avg_loss}} {5}$

which is the calculation done by the code

  • $\begingroup$ Thanks Wang. Could you please explain me in form of some equation or by giving an instance or if you could point to some resources. I am finding it hard to build an intuition of it as I also find the following train_loss in many cases-train_loss += loss.item()*data.size(0) $\endgroup$ – Beginner Sep 23 '19 at 2:07
  • 1
    $\begingroup$ No worries. I've updated with the equations. $\endgroup$ – 1tan Sep 23 '19 at 2:12
  • $\begingroup$ Another noob question, why we multiplied avg_loss with 4?Edit:Oh.. I got it...thanks. $\endgroup$ – Beginner Sep 23 '19 at 2:15
  • 1
    $\begingroup$ When calculating averages, we need its total amount. Multiply avg_loss with 4 to get the total amount of loss in the first 4 batches because avg_loss of the first 4 batches is calculated from total amount divided by 4. $\endgroup$ – 1tan Sep 23 '19 at 2:18
  • $\begingroup$ you are really superb $\endgroup$ – Beginner Sep 23 '19 at 2:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.