We monitor long running industrial engines and we have data series that we want to present on a line chart on a web page. For instance, we have sensors that monitor the oil temperature and pressure on the engine.
There are several other similar data series on the components of the equipment.
The objective is to have a human operator identify deviations in engines, for post-analysis. Our chart will display 24 or 48 hours of engine operation and the operator may identify peaks in temperature or pressure, or on the other measurements.
As such, it is a large amount of data to present on the chart on the web page, and we're starting to hit limitations in several places.
At 24 hours * 3600seconds/hour * 1 data point/second = 86400 data points on the chart.
This amount of points is slowing down the rendering of the web page, and is a lot of data to transfer.
We want to reduce the count of data points presented on the chart, without losing much context. So I ask:
- How can I drop data points without losing much precision?
- What techniques are usually applied in this scenario?
A first (naive) thought was to group them in 5-second windows and only return one data point to represent the 5-second window on the chart;
- should I average the points in the window?
- should I consider the maximum in the window?
Are there other techniques than grouping data points in window, to reduce the loss of meaning for the monitoring?