How can I measure time and memory complexity for a deep learning model?

Question

How can I measure or find the time complexity and memory complexity for a model like VGG16 or Resnet50? Also, will it be different from one machine to another like using GTX GPU or RTX GPU?

Answer 1

Talking of Deep Learning specifically, you will see a lot of research papers that report the following metrics to compare time complexity (Speed of inference) and space complexity (Model size) in their papers -

1. Time complexity in terms of FLOPs (floating-point operations) -

   FLOPs are often used to describe how many operations are required to run a single instance of a given model. The more the FLOPs the more time model will take for inference.

2. Model size in terms of the number of parameters -

   If a perceptron with 5 inputs has 5 trainable weight parameters, then just calculate the total number of trainable parameters in a neural network. This will give you the total number of parameters. This is typically calculated in millions or billions.

For eg -

In the following image from paper, the author compares his approach with other models using these two metrics.

The reason we need both of these metrics is that a model with lesser parameters may have more computations (for eg residual connections in resnet are not trainable yet contribute to computational cost i.e more FLOPs). Vice versa, a model with more parameters might have lesser computations.

Lastly, these metrics are independent of the hardware machines. That means these metrics will scale relatively for the chosen machine for all models. For eg if a model takes 1 second for 10 FLOPs on GPU_1 and takes 2 seconds for the same on GPU_2, then another model with 100 FLOPs will take 10 seconds on GPU_1 and 20 seconds on GPU_2. I am using these numbers just for analogy, don't take them literally.

To calculate FLOPs in PyTorch you can follow this link. And for TensorFlow follow this link.