# Outlier filtering from time series data

I have time series data that I eventually want to cluster after using dimensionality reduction. I am thinking about how to handle outliers. The data has seasonal/periodic patterns. I have tried IQR and similar outlier filtering methods, but they are not optimal for my dataset due to their nature.

Are there some outlier methods that would work well with this kind of data, where I try to find unusual patterns?

I want to apply the outlier filtering to my raw data, even though theoretically, it could be used after dimension reduction or clustering. I have been thinking, for example, comparing daily averages, and if there are extremely high or low days compared to other days, that pattern makes it an outlier. But I think this only filters individual "bad" rows, but not rows where the all-year's pattern is strange.

If you are looking to identify outliers in time series data with seasonal patterns, you can consider using the seasonal decomposition of time series (STL) method. STL decomposes a time series into three components: trend, seasonality, and residuals. The residuals represent the noise or irregular component of the time series.

To identify outliers using STL, you can follow these steps:

1. Decompose the time series into its components using the STL method.
2. Calculate the residuals by subtracting the trend and seasonality components from the original time series.
3. Calculate the z-score for each residual value. The z-score measures how many standard deviations an observation is from the mean. Outliers are typically defined as observations with a z-score greater than a certain threshold (e.g., 3 or 4).
4. Identify the outliers based on the calculated z-scores.

Here is an example code snippet in Python using the statsmodels library:

import numpy as np
import pandas as pd
from statsmodels.tsa.seasonal import seasonal_decompose

# Assuming your time series data is stored in a pandas DataFrame called 'data' with a column named 'value'
# Assuming the time series has a frequency of 1 day

# Decompose the time series

# Get the residuals
residuals = decomposition.resid

# Calculate the z-scores
z_scores = (residuals - np.mean(residuals)) / np.std(residuals)

# Identify outliers
outliers = data[np.abs(z_scores) > 3]

# Print the outliers
print(outliers)


This code uses the seasonal_decompose function from the statsmodels library to decompose the time series into its components. Then, it calculates the z-scores for the residuals and identifies the outliers as the observations with z-scores greater than 3. Finally, it prints the outliers.

You can adjust the threshold for identifying outliers by changing the value in the np.abs(z_scores) > 3 condition.

• Thank you for the comprehensive answer. I tried STL, but the problem for that might be, that there is multiple seasonal components in my data. I also tried MSTL, but it is too computationally hard, as i have thousands of rows and for every row, yearly measurements. My goal is to remove rows that are "acting" differently. Commented Nov 6, 2023 at 7:53