0
$\begingroup$

A retail enterprise I work with with wants to switch from its home-grown time series data analysis and prediction system to something more established and with community support. One unique feature of the existing system is that every bit of input data, from prices to temperature readings, are already converted to integer values in the data warehouse (currently storing over one petabyte of data). The CTO feels strongly that the commodity compute hardware they use will process integer calculations faster (and with greater accuracy in the case of prices, for instance). As an example: A feature with 256 or fewer discrete values can pack much more tightly (8 bits per value) in Streaming SIMD (SSE) registers than the same feature encoded as a 32-bit float. (in such a case, the extended -1.0 to +1.0 value space of the float is not needed and is "wasted" from a certain standpoint)

Certainly, many mathematical operations will produce floating-point results, even if performed on integers (moving averages, etc). Yet these results themselves may be encoded as fixed-point int values or within a scaled int range before being fed to other functions / layers of the system.

My question is: Are there any established frameworks that have better support for end-to-end integer-based features, modeling, and prediction/inference?

Thank you, in advance, for your consideration of this question. If you feel strongly that the premise behind the question is flawed, I'd be intersted in learning why, as well.

Caveat: I'm not asking for a list of preferences. I'm simply not aware of any framework or set of libraries that might fulfill the criteria of my inquiry. My own research via search engines has not yielded any particular guidance, except for scholarly papers locked behind paywalls such as Training and Inference with Integers in Deep Neural Networks . It's not even clear to me whether well-used building blocks in data science such as the c libraries beneath numpy and scipy include versions of most of their functions that accept int types rather than float types, for instance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.