1
$\begingroup$

I have a long univariate time series, and before performing some machine learning models with it, I want to extract as many features as I can from the time series on a rolling-window basis.

As a quick example, for a window of size 10, I would like to calculate statistics like mean and std deviation for the first t=0:9 points in my dataset, and have those two results occupy one row in a some new feature table, and the next row in the table will have mean and std deviation calculated on points t=1:10, and so on and so forth, until the end of the data.

Is there an efficient way to do this in Python?

$\endgroup$
2
$\begingroup$

Yes, there are easy ways to do this in Python. My favourite would be to put the data into a Pandas DataFrame, which has a convenient method called rolling that will cycle over your data in a given frame-size and compute whatever you like on that block.

Let me show you an example - say we start with the following column of data:

In [1]: import pandas as pd                                                     
In [2]: import numpy as np
In [3]: df = pd.DataFrame({"A": np.random.randint(0, 100, (20,)),
                           "B": np.random.randn(20)})

Look at the first 10 rows:

In [4]: df.head(10)
Out[4]: 
     A         B
0   63 -0.003947
1   55  0.442597
2    6  0.684125
3   17  0.968987
4   33 -0.018640
5   50 -0.579558
6   71  0.563125
7   31  1.417384
8    8  0.607813
9   36  0.186146

We can compute the rolling average over each column and save it back to the dataframe like this:

In [6]: df[["rolling_a", "rolling_b"]] = df.rolling(5).mean()
In [7]: df.head(10)
In [9]: df
Out[9]: 
     A         B  rolling_a  rolling_b
0   63 -0.003947        NaN        NaN
1   55  0.442597        NaN        NaN
2    6  0.684125        NaN        NaN
3   17  0.968987        NaN        NaN
4   33 -0.018640       34.8   0.414624
5   50 -0.579558       32.2   0.299502
6   71  0.563125       35.4   0.323608
7   31  1.417384       40.4   0.470260
8    8  0.607813       38.6   0.398025
9   36  0.186146       39.2   0.438982

You might notice that the first 4 rows contain NaN values (Not a Number). This is because the rolling() method will let the mean() method work an a window-size smaller than 5 (in our example). There are a lot of options in the rolling() method that you can experiment with.

You can do the same above for single column of a large dataframe like this:

>>> df["rolling_some_column_name"] = df.some_column_name.rolling(5).mean()

You can also apply just about any function to the rolling frame - not just mean().

| improve this answer | |
$\endgroup$
  • $\begingroup$ Raises error: "None of [Index(['rolling_a', 'rolling_b'], dtype='object')] are in the [columns]" $\endgroup$ – ledawg Apr 18 at 0:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.