1
$\begingroup$

I want to do a platform-free benchmark for some custom ML models. Calculating the elapsed time during making predictions from certain size data is not suitable since I am constantly using different hardware.

This is more of an algorithm complexity question than a data science question. However, I am unsure if the number of parameters linearly corresponds to FLOPS(Floating Point Operations Per Second) values.

I found an algorithm called

The RAM (Random Access Machine) model of computation: Under the RAM model, we measure the run time of an algorithm by counting up the number of steps it takes on a given problem instance. By assuming that our RAM executes a given number of steps per second, the operation count converts easily to the actual run time.

Can we say the definition of this algorithm is similar to FLOPS or the number of parameters?

Edit: I found a related discussion about FLOPS:

https://stackoverflow.com/questions/329174/what-is-flop-s-and-is-it-a-good-measure-of-performance

If you read it you will notice there are lots of contradicting answers. Isnt measuring source usage of ML models an important subject? Why there are no standards for this?

$\endgroup$
3
  • $\begingroup$ My two cents. RAM is more relevant that FLOPS because FLOPS is a hardware-system dependent estimator (my understanding) whereas RAM is looking at steps-based, and you can convert those number of operational steps to any step/second ratio and estimate machine or system specific perormance. $\endgroup$ Nov 4, 2022 at 2:29
  • $\begingroup$ @sconfluentus that makes sense. Do you know any sources for calculating RAM for NNs? I couldn't find much about it. That's why I wanted to ask here. Is RAM the standard of such measurements? Or maybe there is another name for it. $\endgroup$
    – Enes Kuz
    Nov 4, 2022 at 8:42
  • $\begingroup$ You can try this python module, it may work wrapped around your code: pypi.org/project/random-access-machine $\endgroup$ Nov 4, 2022 at 14:06

2 Answers 2

1
$\begingroup$

number of parameters linearly corresponds to FLOPS

In general no, since FLOPS depends not only on # of parameters, but also on the computation required (e.g. model complexity).

For example, say we have 2 parameters, a and b. The number of computation steps vary greatly between $a+b$, $a*b$ and $a^b$.

Another way to think is that it takes less steps to compute $a+b+c$ (3 parameters) than $log(a)+log(b)$ (2 parameters).

$\endgroup$
5
  • $\begingroup$ And is there standard way to calculate FLOP/s $\endgroup$
    – Enes Kuz
    Nov 4, 2022 at 9:03
  • $\begingroup$ FLOP/s of what? It is a metric for computer performance, not directly suitable for ML model. $\endgroup$
    – lpounng
    Nov 4, 2022 at 9:13
  • $\begingroup$ FLOPs = total number of floating point operations to run an algorithm? If this is correct then I need to calculate FLOPs not Flop/s $\endgroup$
    – Enes Kuz
    Nov 4, 2022 at 9:15
  • $\begingroup$ Neither FLOP (# floating point operations) nor FLOPS (floating point operations per second) are suitable for directly evaluating the performance of an algorithm (and ML model), as they both have hardware/software-dependency. That is, for same model, total # operations vary depending on the exact software implementation and hardware setup. $\endgroup$
    – lpounng
    Nov 4, 2022 at 9:27
  • $\begingroup$ This is why we invent the Big O notation for model/algorithm. $\endgroup$
    – lpounng
    Nov 4, 2022 at 9:28
1
$\begingroup$

Isnt measuring source usage of ML models an important subject?

The efficiency of machine learning models is very important. Since machine learning is primarily an empirical field, hardware choices matter.

Why there are no standards for this?

There are standards, one of which is MLPerf. MLPerf has separate categories for training and inference. For inference, MLPerf measures resources in joules. Joules matter when deploying a machine learning model in a resource-constrained environment. Other inference metrics include predictions per second and latency.

$\endgroup$
2
  • $\begingroup$ But aren't these standards hardware-dependent? So how can there be a meaningful comparison if the platform affects the measurements? $\endgroup$
    – Enes Kuz
    Nov 4, 2022 at 17:43
  • $\begingroup$ @EnesKuz here comes the Big O notation again. It is hardware-independent. $\endgroup$
    – lpounng
    Nov 7, 2022 at 1:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.