# What is the best practice to detect constant and nearly constant data over time?

I'm experimenting with the characterization of data over time. I generated some synthesized data with certain periodic patterns over time every 5mins (granularity of 5mins= data generated with the interval of 5 mins). It means that for each hour I generate 12 observations and at the end of the day (24hrs) I have 12*24 = 288 observations\data points over time.

# Set common parameters
samples = 288
num_samples = 288  # number of samples

# Create a time array with 5-minute intervals
t_num = pd.date_range(start='2024-01-01', freq='5T', periods=samples)

# Creating a set of constant data
constant_data = np.full(num_samples, 20)

# Creating a set of nearly constant data
nearly_constant_data = np.random.randint(41, 43, size=(len(t_num)))

# Convert data to a Pandas DataFrame
data = {'datetime': t_num, "constant":constant_data,  "nearly_constant":nearly_constant_data}
df = pd.DataFrame(data)
df.shape #(288, 3)


So now I have a univariate time series including timestamp datetime and some periodic values in the form of constant signal.

I believe that downsampling also has no impact on this analysis.

• df.datetime = pd.to_timedelta(df.datetime, unit='T') ref
• df['datetime'] = pd.to_datetime(df['datetime']) ref

Then I applied resample() to downsample 5mins to 1hour like this post.

resampled_df = (df.set_index('datetime')          # Conform data by setting a datetime column as dataframe index needed for resample
.resample('1H')                 # resample with frequency of 1 hour
.mean()                         # used mean() to aggregate
.interpolate()                  # filling NaNs and missing values [just in case]
)
resampled_df.shape                                # (24, 2)


I used the mean() method because I think the average of each 12 observations within each hour could be a good representative of behavior and has a less negative impact on behavior for Periodic Patterns\behavior Identification.

Now I want to demonstrate raw periodic data and resampled version:

import matplotlib.pyplot as plt
import pandas as pd

fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(15, 4))

#  (nearly-)constant
axes[0].plot(   df['datetime'],  df['constant'],   color='blue')
axes[0].scatter(df['datetime'],  df['constant'],   color='blue', marker='o', s=10)

axes[0].plot(   df['datetime'],  df['nearly_constant'],   color='purple')
axes[0].scatter(df['datetime'],  df['nearly_constant'],   color='purple', marker='o', s=10)
axes[0].set_title(f'constant incl. {len(df)} observations')

# Resample of (nearly-)constant

axes[1].plot(   resampled_df.index,  resampled_df['constant'], color='blue')
axes[1].scatter(resampled_df.index,  resampled_df['constant'], color='blue', marker='o', s=10)

axes[1].plot(   resampled_df.index,  resampled_df['nearly_constant'], color='purple')
axes[1].scatter(resampled_df.index,  resampled_df['nearly_constant'], color='purple', marker='o', s=10)
axes[1].set_title(f'(nearly-)constant (resampled frequency=1H) incl. {len(resampled_df)} observations')

for ax in axes:
ax.set_xticks(selected_ticks)
ax.set_xticklabels(selected_ticks, rotation=90)

plt.show()


Output:

My objective is to detect constant and nearly constant behavior for further characterization of data over time which can be seen in different data resolutions and one needs to downsample also because of the volume of (big-)data. So I'm looking for best practices to downsample and detect onstant and nearly constant behavior.

I find the approach of pandas.DataFrame.rolling() based on this answer but I'm not sure if fits my problem.

also, there is a comment there stating:

"...conduct a hypothesis test. The null hypothesis is that it stays constant, and the alternate hypotheses are for increasing and decreasing. The parameter of the test is the slope of linear regression model, unless there is seasonality, in which case you will need to estimate the trend by time series decomposition. You can do the test in batch or sequentially (cf. sequential hypothesis test). ..."

The closest workaround and mathematic approach I have found so far are:

I'm not sure if the above-mentioned mathematic tools are good enough or best practices to detect (nearly-) constant behavior for the characterization of data over time. And maybe label this type of data and pass it to ML algorithms for classification\clustering tasks?

Time-series: