0
$\begingroup$

In the training loop below, you can see that train_loss represents the loss on the most recent batch.

Whereas the eval_loss is computed on the entire training dataset.

When I print it out, should I be recalculating the train_loss against the entire training split instead of the last batch?

for epoch in range(epochs):
    # Loop over batches.
    for i, batch in enumerate(samples_train['features']):
        
        train_preds = model(samples_train['features'][i])
        train_loss = loss_fn(train_preds, samples_train['labels'][i])

        optimizer.zero_grad()
        train_loss.backward()
        optimizer.step()
        
    # Only the training data is batched.
    eval_preds = model(samples_evaluate['features'])
    eval_loss = loss_fn(eval_preds, samples_evaluate['labels'])
    
    print(f"Epoch: {epoch}, Train_Loss: {float(train_loss)},  Eval_Loss: {float(eval_loss)}")
$\endgroup$
1
  • $\begingroup$ hmm now i am thinking about undersized batches and leaning toward the whole split $\endgroup$ – Kalanos Apr 6 at 23:28
1
$\begingroup$

The losses should be calculated over the whole epoch (i.e. the whole dataset) instead of just the single batch. To implement this you could have a running count which adds up the losses of the individual batches and divides it by the number of batches after the epoch ends.

$\endgroup$
1
  • $\begingroup$ Nice yeah that would be much more efficient than loading the whole dataset again. $\endgroup$ – Kalanos Apr 7 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.