How to estimate the time required time to train a model, given feature shape, CPU/GPU sepcs and type of model


There is no answer to that question because besides model parameters, hardware specs and dataset size, there is also the question of the difficulty of the problem, which is not quantifiable.

An easily-solvable problem can require fewer epochs for the model to sufficiently converge. Take for example two image classification tasks:

  • The first is ILVRC which contains around 1m images.
  • Another dataset that contains 1m red and green images (just pure red and pure green).

Both datasets have the same image dimensions, the same number of images and will be used to train the same model on the same computer. Even though these tasks seem identical, the latter is much easier to solve and will require only a few iterations (i.e. a few seconds). The former is harder and could require 2-3 weeks.

Due to this huge difference caused, solely, on a non quantifiable characteristic of the dataset, there is currently no way to tell how long a model will require to train.

Furthermore, due to the stocahasticity of the training process, two identical models might take different times to converge, even on the same dataset (due to different initial conditions).

Note: what you can estimate is the time it takes for each epoch if you know the model (number of parameters, choice of certain hyperparameters), the dataset (features, samples) and the hardware.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.