How can I measure or find the time complexity and memory complexity for a model like VGG16 or Resnet50? Also, will it be different from one machine to another like using GTX GPU or RTX GPU?
Talking of Deep Learning specifically, you will see a lot of research papers that report the following metrics to compare time complexity (Speed of inference) and space complexity (Model size) in their papers -
Time complexity in terms of FLOPs (floating-point operations) -
FLOPs are often used to describe how many operations are required to run a single instance of a given model. The more the FLOPs the more time model will take for inference.
Model size in terms of the number of parameters -
If a perceptron with 5 inputs has 5 trainable weight parameters, then just calculate the total number of trainable parameters in a neural network. This will give you the total number of parameters. This is typically calculated in millions or billions.
For eg -
In the following image from paper, the author compares his approach with other models using these two metrics.
The reason we need both of these metrics is that a model with lesser parameters may have more computations (for eg residual connections in resnet are not trainable yet contribute to computational cost i.e more FLOPs). Vice versa, a model with more parameters might have lesser computations.
Lastly, these metrics are independent of the hardware machines. That means these metrics will scale relatively for the chosen machine for all models. For eg if a model takes 1 second for 10 FLOPs on GPU_1 and takes 2 seconds for the same on GPU_2, then another model with 100 FLOPs will take 10 seconds on GPU_1 and 20 seconds on GPU_2. I am using these numbers just for analogy, don't take them literally.
To calculate FLOPs in PyTorch you can follow this link. And for TensorFlow follow this link.
$\begingroup$ appreciate your help, thanks . but excuse me how can I measure flop for the model. and please if I trained, for example, a captioning model already and have h5 file at epoch 20 . cam I measure the time it took by multiplying 20* time in minutes that every epoch took ? $\endgroup$– LeiDec 2, 2021 at 9:52
$\begingroup$ You're Welcome. I have updated the answer! And yes you can also do it that way. This type of time calculation will be specific to your machine. $\endgroup$ Dec 2, 2021 at 12:58
1$\begingroup$ You just need to call the function and pass the path in arguments. I think that number means ~ 12 x 10^6 FLOPs (as shown in the image). That is how many computational operations your model needs. $\endgroup$ Dec 2, 2021 at 18:43