1
$\begingroup$

I am trying to model an image classification problem using convolution neural network. I came across a code on Github in which I am not able to understand the meaning of following line for loss calculation in the training loop.
I am omitting most of the detail and only placing the relevent code-

for batch_idx, (data, target) in enumerate(final_train_loader):
     loss = criterion(output,target)
     #Idea behind the below line
     train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))

Cross-entropy loss function is being used here.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The line you're asking about

train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))

is basically calculating the average train_loss for the finished batches

To illustrate, suppose 4 batches have been done (with average loss named avg_loss) and current is calculated from 5th batch (with loss named new_loss)

The new average loss is from

$\frac {4 \times \text{avg_loss} + \text{new_loss}} {5}$

This is exactly the same as

$\text{avg_loss} + \frac {\text{new_loss} - \text{avg_loss}} {5}$

which is the calculation done by the code

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks Wang. Could you please explain me in form of some equation or by giving an instance or if you could point to some resources. I am finding it hard to build an intuition of it as I also find the following train_loss in many cases-train_loss += loss.item()*data.size(0) $\endgroup$
    – Beginner
    Commented Sep 23, 2019 at 2:07
  • 1
    $\begingroup$ No worries. I've updated with the equations. $\endgroup$
    – 1tan Wang
    Commented Sep 23, 2019 at 2:12
  • $\begingroup$ Another noob question, why we multiplied avg_loss with 4?Edit:Oh.. I got it...thanks. $\endgroup$
    – Beginner
    Commented Sep 23, 2019 at 2:15
  • 1
    $\begingroup$ When calculating averages, we need its total amount. Multiply avg_loss with 4 to get the total amount of loss in the first 4 batches because avg_loss of the first 4 batches is calculated from total amount divided by 4. $\endgroup$
    – 1tan Wang
    Commented Sep 23, 2019 at 2:18
  • $\begingroup$ you are really superb $\endgroup$
    – Beginner
    Commented Sep 23, 2019 at 2:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.