1
$\begingroup$

Before starting a new machine learning side project, it would be very useful to estimate how long it will take to run 1, 10, 100, 1k epochs. A crude estimate is more than sufficient (i.e. 1 epoch would take 1 second, 10 seconds, 1 minute, 1 hour, etc..).

Given the variables below, can you recommend any heuristics that could provide an estimate?

  1. Problem type (e.g. Image Segmentation)
  2. Model type (e.g. PyTorch Unet)
  3. Dataset (e.g. 10k images, 512x512)
  4. Compute (e.g AWS p2.xlarge)
  5. Library (e.g. PyTorch)

Is an empirical method (e.g train on smaller subsets of the data and scale accordingly) a better approach to solving this problem?

$\endgroup$
0
$\begingroup$

The best way to know is to train two epochs. The first epoch often takes longest, because data loading takes place with some caching.

The second epoch will give you an accurate time for each epoch.


Things to help the guesses that you might include in your question:

  • What are the specifications of p2.xlarge?
  • Where are your images stored? on the instance EBS storage? s3 storage? A separate file-system?
  • In which format is your data stored?
  • Perhaps the number of parameters in the model, but that can also be misleading!
$\endgroup$
5
  • $\begingroup$ "The first epoch often takes longest, because data loading takes place with some caching.", will it be a epoch or first batch? For me its first batch only as it can be very costly to keep everything in memory. $\endgroup$ – vipin bansal Apr 21 '20 at 4:49
  • $\begingroup$ If you have a big dataset, a big model, and are using GPUs, it is the first epoch. The first batch begins when data for a single batch has been loaded into memory. Frameworks like PyTorch and TensorFlow use multiple threads to continue loading data from disk (e.g. SSD) into memory (RAM). After the first epoch, all one-time work has been done, as the dataset has been completely iterated (that's the definition of an epoch). All subsequent epochs re-use any cached data and are only ever faster (assuming no costly scheduling during training). $\endgroup$ – n1k31t4 Apr 21 '20 at 6:44
  • $\begingroup$ Thanks @vipinbansal, perhaps with enough experiments it should be possible to derive a reasonably accurate heuristic. [At least for a reasonably narrow set of compute options, popular models and ML frameworks] $\endgroup$ – Tommy Apr 21 '20 at 16:12
  • $\begingroup$ Very true. I'm thinking in a way somehow model reaches in a stage where the variables values are in integer. Since integer processing is lighter in comparison to float value, it's difficult to come on conclusion on the basis of few iteration. $\endgroup$ – vipin bansal Apr 21 '20 at 16:55
  • $\begingroup$ The variables (weights) of the neural network will never reach integer values; unless you have trained using quantisation, which itself has to be done very deliberately. Otherwise, even if values are coincidentally integers, they are still stored as the data type they always were, so float32 (by default in the aforementioned frameworks). $\endgroup$ – n1k31t4 Apr 21 '20 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.