0
$\begingroup$

I'm experimenting with the characterization of data over time. I generated some synthesized data with certain periodic patterns over time every 5mins (granularity of 5mins= data generated with the interval of 5 mins). It means that for each hour I generate 12 observations and at the end of the day (24hrs) I have 12*24 = 288 observations\data points over time.

# Set common parameters
samples = 288
num_samples = 288  # number of samples

# Create a time array with 5-minute intervals
t_num = pd.date_range(start='2024-01-01', freq='5T', periods=samples)

# Creating a set of constant data
constant_data = np.full(num_samples, 20)

# Creating a set of nearly constant data
nearly_constant_data = np.random.randint(41, 43, size=(len(t_num)))

# Convert data to a Pandas DataFrame
data = {'datetime': t_num, "constant":constant_data,  "nearly_constant":nearly_constant_data}
df = pd.DataFrame(data)
df.shape #(288, 3)

So now I have a univariate time series including timestamp datetime and some periodic values in the form of constant signal.

I believe that downsampling also has no impact on this analysis.

  • df.datetime = pd.to_timedelta(df.datetime, unit='T') ref
  • df['datetime'] = pd.to_datetime(df['datetime']) ref

Then I applied resample() to downsample 5mins to 1hour like this post.

resampled_df = (df.set_index('datetime')          # Conform data by setting a datetime column as dataframe index needed for resample
                  .resample('1H')                 # resample with frequency of 1 hour
                  .mean()                         # used mean() to aggregate
                  .interpolate()                  # filling NaNs and missing values [just in case] 
                )
resampled_df.shape                                # (24, 2)

I used the mean() method because I think the average of each 12 observations within each hour could be a good representative of behavior and has a less negative impact on behavior for Periodic Patterns\behavior Identification.

Now I want to demonstrate raw periodic data and resampled version:

import matplotlib.pyplot as plt
import pandas as pd

fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(15, 4))

#  (nearly-)constant
axes[0].plot(   df['datetime'],  df['constant'],   color='blue')
axes[0].scatter(df['datetime'],  df['constant'],   color='blue', marker='o', s=10)

axes[0].plot(   df['datetime'],  df['nearly_constant'],   color='purple')
axes[0].scatter(df['datetime'],  df['nearly_constant'],   color='purple', marker='o', s=10)
axes[0].set_title(f'constant incl. {len(df)} observations')

# Resample of (nearly-)constant

axes[1].plot(   resampled_df.index,  resampled_df['constant'], color='blue')
axes[1].scatter(resampled_df.index,  resampled_df['constant'], color='blue', marker='o', s=10)

axes[1].plot(   resampled_df.index,  resampled_df['nearly_constant'], color='purple')
axes[1].scatter(resampled_df.index,  resampled_df['nearly_constant'], color='purple', marker='o', s=10)
axes[1].set_title(f'(nearly-)constant (resampled frequency=1H) incl. {len(resampled_df)} observations')

for ax in axes:
    ax.set_xticks(selected_ticks)
    ax.set_xticklabels(selected_ticks, rotation=90)


plt.show()

Output: img

My objective is to detect constant and nearly constant behavior for further characterization of data over time which can be seen in different data resolutions and one needs to downsample also because of the volume of (big-)data. So I'm looking for best practices to downsample and detect onstant and nearly constant behavior.

I find the approach of pandas.DataFrame.rolling() based on this answer but I'm not sure if fits my problem.

also, there is a comment there stating:

"...conduct a hypothesis test. The null hypothesis is that it stays constant, and the alternate hypotheses are for increasing and decreasing. The parameter of the test is the slope of linear regression model, unless there is seasonality, in which case you will need to estimate the trend by time series decomposition. You can do the test in batch or sequentially (cf. sequential hypothesis test). ..."

The closest workaround and mathematic approach I have found so far are:

I'm not sure if the above-mentioned mathematic tools are good enough or best practices to detect (nearly-) constant behavior for the characterization of data over time. And maybe label this type of data and pass it to ML algorithms for classification\clustering tasks?


Time-series:

$\endgroup$

1 Answer 1

1
$\begingroup$

You attempt to check whether data is (nearly) constant, and for me, it appears unclear and ambiguous. I came up with an idea that you could try to inverse your task and try to detect phenomena that cause that data to be not constant. I think that you could establish such issues and it would tell you not only that your series is not constant, but also what is happening.

The list of issues that came to my mind causing that the series is not constant (and how to detect it):

  • outliers (e.g. isolation forest)
  • changepoints (e.g. Bayesian ChangePoint Detection)
  • trends (e.g. statistical testing of a local regression coefficient)
  • turning points (e.g. Bry-Boschan routine or I can recommend the algorithm I worked on)

It seems to me that such a way of thinking will be more robust in your problem.

$\endgroup$
1
  • $\begingroup$ Thanks for your input to look at the problem from another angle. That'd be very complicated then. I was hoping to do it using some straightforward math tools similar to what I found (i.e., AAD, L1-variance). I'm interested in a quick way to detect and pre-filter those nearly-constant observations from data and work over non-(nearly-)constant. $\endgroup$
    – Mario
    Jan 25 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.