0
$\begingroup$

i need to analyse and later try to improve (integrate a filter) for measurement data that i compare to accurate reference data with python.

First i want to calculate the mean offset and the standard deviation of the measurement data overall, during halt, increasing and decreasing state.

How can i automatically detect and mark sections in which the reference system data stays constant, increases or decreases and use this information later for the statistical analysis.

I already tried to write a simple algorithm myself but the problem is though the reference data is given precise enough so you can calculate the difference of two neighbouring data points and it is zero if no change takes place this case is also in the majority during sections of change (increase, decrease). So the algorithm can't really mark the state of the data points reliable.

I already researched a little bit and here are some keywords that seemed interesting for my problem: time series analysis, anomaly detection, novelty detection, change point detection, structural changes, One-Class Support Vector Machines

So my question would be if sb. could give me a pointer towards the right direction (correct method, python package, tutorial, example) so i can solve this problem myself.

$\endgroup$
  • 1
    $\begingroup$ You want to conduct a hypothesis test. The null hypothesis is that it stays constant, and the alternate hypotheses are for increasing and decreasing. The parameter of the test is the slope of linear regression model, unless there is seasonality, in which case you will need to estimate the trend by time series decomposition. You can do the test in batch or sequentially (cf. sequential hypothesis test). Welcome to the site and good luck! $\endgroup$ – Emre Jun 4 '18 at 20:43
1
$\begingroup$

First i want to calculate the mean offset and the standard deviation of the measurement data overall, during halt, increasing and decreasing state.

I would recommend checking windows of your dataset (let's say 10 samples in a row). A very convenient method to do that is pandas.DataFrame.rolling(), please take a look at it :)

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thank you. I'll try this approach but i am still in the process of learning pandas better. $\endgroup$ – Vash Jun 7 '18 at 9:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.