How to drop points from a data series for presentation?

Question

We monitor long running industrial engines and we have data series that we want to present on a line chart on a web page. For instance, we have sensors that monitor the oil temperature and pressure on the engine.

There are several other similar data series on the components of the equipment.

The objective is to have a human operator identify deviations in engines, for post-analysis. Our chart will display 24 or 48 hours of engine operation and the operator may identify peaks in temperature or pressure, or on the other measurements.

As such, it is a large amount of data to present on the chart on the web page, and we're starting to hit limitations in several places.

At 24 hours * 3600seconds/hour * 1 data point/second = 86400 data points on the chart.

This amount of points is slowing down the rendering of the web page, and is a lot of data to transfer.

We want to reduce the count of data points presented on the chart, without losing much context. So I ask:

• How can I drop data points without losing much precision?
• What techniques are usually applied in this scenario?

A first (naive) thought was to group them in 5-second windows and only return one data point to represent the 5-second window on the chart;

• should I average the points in the window?
• should I consider the maximum in the window?

Are there other techniques than grouping data points in window, to reduce the loss of meaning for the monitoring?

Answer 1

Indeed this problem isn't a very simple one to deal with, despite looking very easy to conceptualise. There exist a certain number of techniques for "reducing" the number of points of timeseries, one being called "downsampling".

A little litterature : https://skemman.is/bitstream/1946/15343/3/SS_MSthesis.pdf

hope this helps,

Cheers

Answer 2

It depends on the humanly acceptable frequency: If the operator can react on a minute basis, then the max value during one minute is enough.

However, if you have to react quickly according to recent values, you can scale your record frequency based on the age of the data.

For instance, very recent records (from 0 to 5 minutes) would have a frequency of 1Hz, then recent ones (from 5 minutes to 1 hour) would have a frequency of 0.15Hz (1 record per minute), and the remaining data 0.003Hz (1 record every 5 minutes).

An interesting option could be to increase the frequency just around the peaks.