3
$\begingroup$

I have a tabular raw data from sensors with associated label and i want to extract the time series features like mean,max,min and std from the data all the sensor data and form another table or export to csv file so that i can do classification task on that data.

Data table

enter image description here

$\endgroup$
3
  • 1
    $\begingroup$ Could you be more precise problem as what you problem is and what tools/ languages you are planning to use? $\endgroup$
    – El Burro
    Commented May 29, 2018 at 11:40
  • $\begingroup$ I am using python with data table like data1_mean,data1_max,data1_min etc also how to transform the label. $\endgroup$
    – rosy
    Commented May 29, 2018 at 19:07
  • $\begingroup$ you can look at the tsfresh repository on github. It extract time series features from the sensor logs $\endgroup$ Commented Feb 23, 2019 at 23:49

4 Answers 4

2
$\begingroup$

For clarification: mean,max,min,std are not "time series features", they are data features in general.

Assuming that you want to do it in python, you should take a look at pandas.DataFrame class. Once you initialize a Dataframe object with your tabular data, you can call its methods DataFrame.min(), DataFrame.max(), DataFrame.mean(), DataFrame.std() for your purpose.

You can insert all these calculated characteristics into a new DataFrame and thereafter call Dataframe.to_csv() to export them in a csv file.

$\endgroup$
5
  • $\begingroup$ Thanks but i want a rolling window of 10 seconds and with 50 % data overlapping also what about the label. $\endgroup$
    – rosy
    Commented May 29, 2018 at 19:05
  • $\begingroup$ This was not in the original question. Anyway, you can always create a for-loop going through all samples in batches of 10s with 50% overlapping, then assign the batch in a DataFrame object (inside the loop) and then call the appropriate methods for the batch. It will work smoothly. $\endgroup$
    – pcko1
    Commented May 29, 2018 at 19:17
  • $\begingroup$ also check this pandas.pydata.org/pandas-docs/stable/generated/… :) $\endgroup$
    – pcko1
    Commented May 29, 2018 at 19:37
  • $\begingroup$ Thanks but if i use loop across the dataset then what about the label corresponding to that,should i use the label with highest frequency on that window. $\endgroup$
    – rosy
    Commented May 29, 2018 at 19:41
  • $\begingroup$ Since the label is binary, you cannot "average" it into a real number because it would not make sense. Your best choice is to use the "majority vote" of the window, which is the label with the highest frequency as you mentioned. $\endgroup$
    – pcko1
    Commented May 29, 2018 at 19:44
3
$\begingroup$

You can also use an open source python library called 'tsfresh' (https://tsfresh.readthedocs.io/en/latest/) to extract time series features

$\endgroup$
2
$\begingroup$

Perhaps you need to look at this self-contained blogpost on Machine Learning with Signal Processing Techniques on how to prepare your time series data and extract useful statistical estimate and feature for machine learning models. At the end an example is given for classification. I found it super useful and straightforward.

Somewhere in the middle of the post, this great method for the Detection of peaks in data is introduced as well.

$\endgroup$
0
$\begingroup$

I don't have enough reputation to leave a comment, but could you please provide some sample data so that we can help you better?

When you say mean, max, min, are you trying to aggregate multiple rows of data on a date column with these functions? Or, do you have a timespan/ datetime/ timestamp column that you want to use?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.