0
$\begingroup$

For a project, I'm trying to predict leaks in a network. The network consists of nodes connected by links. What I have are several 'scenarios' where each scenario has a leak present at a different node (it is also possible that there isn't a leak).

I have pressure data for each node, as well as the flow data for all the links. I also have the targets for all the scenarios (0/1, depending on if there's a leak or not). The data can be represented as below:

Scenario Node 1 Node 2 Output
1 [69.887, 70.383, ...] [71.875, 72.421, ...] 0
2 [60.358, 61.530, ...] [76.840, 77.389, ...] 1

In case there is a leak present, during the time the leak exists, the pressure takes a massive dip and the flow shoots up. This is applicable to most nodes and links in the system, depending on where they're located.

Pressure when leak exists: Pressure when leak exists

Flow when leak exists: enter image description here

My initial idea was to represent all the data at a particular node as a single value using either standard deviation or mean, but that would result in the loss of the data's trends.

I wanted to see how I could possibly train a model while either keeping the data as is or transforming it in some way. I was considering using a Random Forest classifier for this, but I'm open to suggestions.

$\endgroup$
0
$\begingroup$

A very simple approach is to build a detector with a sliding window on which you compute standard deviation. For each slide compute the mean and you can test if the next point after the window is far from mean more than a number of standard deviations. This should work because if the window is not very large it will adapt to the local mean and also if the functions are relatively constant on windows and spikes are large, as they look from your graphs

$\endgroup$
2
  • $\begingroup$ Thank you so much for the answer! What you suggested is something that I'd considered but rejected since its complexity would be quite high. For systems that consist of a higher number of nodes, it would be time-consuming and quite computationally expensive. $\endgroup$
    – Hridik P
    Apr 8 at 2:31
  • $\begingroup$ Mean and variance of a rolling window can be computed in linear time. Just search for such kind of implementations, you are not alone with that and you can find it. Less than linear time is close to impossible to have a reliable outlier detection. $\endgroup$
    – rapaio
    Apr 8 at 5:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.