Is number of Parameters a sufficient benchmark for measuring how much resource the end model will use?

Question

I want to do a platform-free benchmark for some custom ML models. Calculating the elapsed time during making predictions from certain size data is not suitable since I am constantly using different hardware.

This is more of an algorithm complexity question than a data science question. However, I am unsure if the number of parameters linearly corresponds to FLOPS(Floating Point Operations Per Second) values.

I found an algorithm called

The RAM (Random Access Machine) model of computation: Under the RAM model, we measure the run time of an algorithm by counting up the number of steps it takes on a given problem instance. By assuming that our RAM executes a given number of steps per second, the operation count converts easily to the actual run time.

Can we say the definition of this algorithm is similar to FLOPS or the number of parameters?

Edit: I found a related discussion about FLOPS:

https://stackoverflow.com/questions/329174/what-is-flop-s-and-is-it-a-good-measure-of-performance

If you read it you will notice there are lots of contradicting answers. Isnt measuring source usage of ML models an important subject? Why there are no standards for this?

Answer 1

number of parameters linearly corresponds to FLOPS

In general no, since FLOPS depends not only on # of parameters, but also on the computation required (e.g. model complexity).

For example, say we have 2 parameters, a and b. The number of computation steps vary greatly between $a+b$, $a*b$ and $a^b$.

Another way to think is that it takes less steps to compute $a+b+c$ (3 parameters) than $log(a)+log(b)$ (2 parameters).

Answer 2

Isnt measuring source usage of ML models an important subject?

The efficiency of machine learning models is very important. Since machine learning is primarily an empirical field, hardware choices matter.

Why there are no standards for this?

There are standards, one of which is MLPerf. MLPerf has separate categories for training and inference. For inference, MLPerf measures resources in joules. Joules matter when deploying a machine learning model in a resource-constrained environment. Other inference metrics include predictions per second and latency.