How can I detect events on fuel tank

Question

Hi guys. I'm not a data analyst and I need some direction in this. I'm looking for a way to know the events of the fuel during a range of time, could be a day or a month, etc. If the consumption was like the picture above would be easy.

The problem is that the data I have is like this picture.

I need to take the fueling and the possible thefs, that considering that I have false data, that could go up or down. I can see that those abrupt changes are fake because it always comes to the original value, the main problem is that I don't know how long they are gonna stay fake. Could be just one point or n points of bad points.

So, how can I detect these events (fueling and possible theft) whitout counting the errors as events?. What algorithms or formulas can I apply here?

Answer 1

Welcome to SO! It looks like you have a time series problem. Typically the first step when dealing with time series is to consider the difference. Let us define $f(t)$ as the fuel level at time $t$. You would want to calculate $diff_{\text{fuel}}(t) = f(t) - f(t - 1)$.

After this step you will likely see that the spikes that you identify as bad data are outliers. You could detect these by for example looking at all the data below and above your 2.5 percentile or 5 percentile. Typically this requires some careful analysis work to ensure that you do not delete too much. Once you have identified what the best workable percentile is you can could Winsorize your data.

Lastly, you would look at the data points in the bottom of your resulting distribution. These will likely be the points you identify as theft.

I hope this helps.

Answer 2

So as my comment suggested, I think there is a relatively easy way to solve this. That is with a Fourier transform. As I now recall, removing part of a time based signal is easy when representing the signal in the frequency domain. Because all that is necessary is to apply a filter to the signal to remove unwanted components of the signal. In your case those spikes (abrupt changes) will look like high frequency signals in the frequency domain. So in theory, you should be able to apply a low pass filter to the transformed data to get rid of the spikes ( fakes ), then transform the filtered signal back into the time domain. Then just like that you have gotten rid of all those abrupt changes in your data.

To reiterate.

1. Transform existing signal (your data) into the frequency domain, using some variation of the Fourier transform.

2. Apply a low pass filter to the transformed signal (your transformed data). Note you will need to pick a suitable cutoff frequency for your low pass filter. This should be relatively simple as most of your data is changing very slowly.

3. Apply the inverse of the Fourier transform to your transformed data to bring it back to the time domain. Your spike should be gone in your data now.

This is a very simple process and should only take about 3~4 lines of code in any language with a DSP library.