I'm training a multi-task, multi-label neural network. I am attempting to tune the architecture and am having some trouble interpreting the learning curves. Particularly, when I look at the learning curves for the best performing architectures, the training loss is quite small after the first epoch and decreases very gradually. The validation loss starts very large and decreases much more rapidly than the training loss. There are two things I don't understand. 1) How does a small change in training loss result in a large change in validation loss? 2) Are these curves indicative of over or underfitting?

enter image description here

EDIT: I made a mistake in the training loss calculation which makes these learning curves irrelevant

  • 1
    $\begingroup$ With that difference in scale, you may have a problem in how you compute those losses... $\endgroup$
    – noe
    Commented Jan 31, 2023 at 22:08
  • $\begingroup$ You're absolutely right, I was calculating the training loss incorrectly. Silly mistake, thank you! It now looks much more reasonable $\endgroup$
    – tensormoby
    Commented Jan 31, 2023 at 22:13


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.