In my problem, deep learning is trained on on 100GB data.


In order to monitor the training error versus epochs, we need to evaluate the training loss on our datasets, but does that mean we have to forward pass 100GB data in order to get the exact metric? What do people do in reality?

You don't want evaluate 100 GB data every 100 epochs....Maybe some random sampling?

  • 1
    $\begingroup$ I'm confused: isn't training itself much more computer-intensive than evaluating? My intuition is that if a model can be trained with 100GB data, the evaluation part is negligible. Am I missing something? $\endgroup$
    – Erwan
    Commented Jan 8, 2021 at 23:16
  • $\begingroup$ @Erwan, when you are training 100GB data, you are not feeding 100 GB into memory. You only take a batch of it, say 1MB amount of data into memory to compute the gradient. $\endgroup$ Commented Jan 9, 2021 at 5:22
  • 1
    $\begingroup$ In theory the evaluation doesn't need the full data in memory either, but the option of "evaluation by batch" might not be provided through a standard API. A simple solution would be to randomly select a subset of the training set for evaluation, the performance will be almost exactly the same with a large amount of data. $\endgroup$
    – Erwan
    Commented Jan 9, 2021 at 8:01


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.