I deploy machine learning models (typically GPU) to a variety of environments. I work sort of at the edge of ML R&D and devops, so I am really big into reproducibility, and one thing that drives me nuts is when models output similar but not byte-for-byte identical values, frustrating any hash-based automated testing. For example, here is a score from the same sample, inference model, code, container image, etc, but one is on a Titan, one is on an RTX 2080.

Titan X  = 0.9887396097183228
RTX 2080 = 0.9887396693229675

That's a relative error of 6.0e-08 or about 60 parts-per-billion. Obviously, this is the "same number", well within the weirdness of IEEE 754 and GPU processing.

I'd like to truncate the output of my algos, as this simplifies the job of automated testing, but the question then becomes, how to round to achieve this, and how much precision? Sounds simple, but take for example this note on numpy.around

Results may also be surprising due to the inexact representation of decimal fractions in the IEEE floating point standard [R9] and errors introduced when scaling by powers of ten.

One part-per-million (for typically 0.0-1.0 confidence values) seems reasonable but I'm not sure if there are any subtle gotchas with this approach.


I'm not expert in this but as far as I know the proper way to test for equality modulo floating point imprecision is to compare the differences of the two values, i.e. instead of:

trunc(a) == trunc(b) 

one would do:

abs(a-b) <= epsilon

where epsilon is the constant which represents an acceptable difference, e.g. $10^{-6}$. Of course this requires to read the two values and calculate the difference, instead of simply comparing one against the other.

  • $\begingroup$ abs(a - b) < epsilon is currently what I do when I am explicitly checking if two values are congruent, usually with e=1e-6. After a bit of digging, I found numpy.isclose, which is very similar but also provides a relative tolerance. It has a big honkin caveat about the default absolute tolerance of 1e-8 for values near zero. Incidentally this is the same magnitude as my inter-gpu difference. $\endgroup$ Dec 18 '19 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.