Note: With a meter variable a timestamped value is the sum of all previous differences plus a difference to the most recent value. Think of a electricity meter counting the use of energy.

The goal here is to perform some form of regression (e.g. a Random Forest method) on a data series of a meter variable and then use the resulting model to fix gaps in the data series and possibly do further analysis on the data, for example removing noise from a faulty sensor.

However the possible patterns in the data are likely periodic on timescales smaller than the entire series. Thus we're not interested in modeling the sum of the values. We transform the data by calculating $$\delta_i=\frac{v_i-v_{i-1}}{t_i - t_{i-1}}$$ and then perform the regression on $\delta$.

With the generated model $m(t)$ we can calculate a missing value $v_i$ as $$v_i = v_{i-1} + (t_i - t_{i-1})\cdot m(t).$$

Easy, right?

But now let's throw in some serious noise. Depending on the chosen regression method this doesn't matter much and only makes the predictions worse, but still useful. But what if we want to calculate a missing $v_i$ and $v_{i-1}$ happens to be some bogus value? The model doesn't mind, but for the calculation of $v_i$ we have to assume that $v_{i-1}$ is correct.

Is there a way around that? Is it possible to calculate missing values using only (time-)local information?


Your "missing value" formula mentions $v_{i-1}$, and you seem to be suggesting that you'd prefer not to use that. Ok, fair enough. So use an estimate, $\hat{v}_{i-1}$, which is based on weighted regression of $v_{i-1}$ and several preceding values. Give greater weights to more recent values.

You may wish to implement a classifier that labels each reading as "good" or "bogus". Part of that is straightforward, as you already have a "missing" label for some of your samples. The classifier would additionally discard readings that swing implausibly high, or that show physically impossible behavior like having negative derivative.

You don't mention the generative source of bogus readings. If they are due to noise events on the digital channel used for obtaining each reading, then they don't affect the data source and will soon average out. Here is another technique. Suppose you're willing to suffer higher readout latency in exchange for numeric accuracy. Delaying by one sample would let you compute $\hat{v}_{i}$ as median of three values: $v_{i-1}$, $v_{i}$, $v_{i+1}$. Similarly, you could delay by two and find median of five values, or by three with median of seven values.

  • $\begingroup$ Thank you! I was thinking about using an estimate calculated from previous values, but wasn't sure how exactly. Your answer helped me to figure it out. Edit: The generative source are both the sensors themselves not working correctly and some transmitters having the annoying behavior of sending values of "0" when the sensors are not responding, instead of just not recording a value at all. I'm afraid a classifier is not a viable solution. The whole process will run every day on thousands of data series and any kind of handcoded training data classification will be very prone to overfitting. $\endgroup$ Dec 15 '18 at 12:44
  • $\begingroup$ Well, classifying "0" as "missing" seems an easy thing to add to your analysis. $\endgroup$
    – J_H
    Dec 15 '18 at 12:50
  • $\begingroup$ The zeros should be easy to remove, yes. But the bogus data might take other values depending on the technical specifications of the sensor and not all data series will be meter variables (the process needs to handle both). E.g. a temperature of 9999°C is definitely wrong, but an airflow sensor might measure 1999m³/h either because the pump is running at full capacity or because the airflow sensor is not rated for values higher than that. $\endgroup$ Dec 15 '18 at 13:03
  • $\begingroup$ That suggests adding two more analyses to your sanity checker: Consider turning constant reading (especially a "popular" constant) into NaN, and also look for clipped values. If a diurnal cycle involves long intervals of positive derivative, and negative derivative, and exactly zero derivative without any noise, then perhaps it corresponds to a maxed out airflow sensor. In the end, rather than guessing, you will need physical groundtruth from test runs in order to properly interpret your sensor outputs. $\endgroup$
    – J_H
    Dec 17 '18 at 14:16
  • $\begingroup$ Another technique that is very helpful is exploiting co-located (or nearby) sensors. If the same physical environment stimulates both sensors, you would expect similar results. Each fan turn-on event should produce a "wave" of new readings for the several temperature or air flow sensors located downstream from it. If sensor locations are unknown, they might be inferred by looking for such a wave event percolating through them. Installing a temporary sensor can help with calibration. So can swapping a pair of production sensors, e.g. by exchanging sensors in the "cool" and "warm" locations. $\endgroup$
    – J_H
    Dec 17 '18 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.