Anomaly detection k-means in Time Series

Question

I'm trying to use k-means to detect anomalies in the Amount column. I have the following part of my dataset:

8   2018-09-06  -2760.14
9   2018-10-04  -1149.73
10  2018-11-07  -1551.41
11  2018-12-06  -1208.17
12  2019-01-07   -244.02
13  2019-02-06    111.00
14  2019-03-06    139.24
15  2019-04-04 -27315.10
16  2019-05-07 -30326.58
17  2019-06-06  -1633.44

I used 2 as number of clusters and I get the following centroids: [[ -237.045] [-28820.84 ]]
In my case the anomalies are rows 15 and 16 but now they form their own cluster so the distance to their centroid will be insignificant and thus not an anomaly
My question is, how can I detect the anomalies if they form their own cluster?

Answer 1

K-means is sensitive to extreme values, such as outliers.

This is because A) it assigns every point to a cluster, but outliers shouldn't be clustered, and B) it minimizes the sum-of-squares, which puts more weight on far instances such as outliers. Larger errors get even larger, and hence k-means tries to minimize the cost of outliers by making them cluster centers.

It's the wrong tool for this problem, you don't have a clustering problem.

Answer 2

I don't suggest to use k-Means clustering for outlier detection. You can check the outlierness of observations by taking the standardized distance of each observation from the series' trend.

Alternatively, you can use DBSCAN algorithms: they are clustering models specifically designed to isolate outliers. I don't know if it makes sens on time series data. I'd go for the first method.

Answer 3

This seems like a z-score problem. Try this.

import pandas as pd
from scipy import stats
import numpy as np

df = pd.DataFrame({'foo': ['43349', '43377',    '43411',    '43440',    '43472',    '43502',    '43530',    '43559',    '43592',    '43622'],
                       'baz': [-2760.14,    -1149.73,   -1551.41,   -1208.17,   -244.02,    111,    139.24, -27315.1,   -30326.58,  -1633.44]})
# print(df)

def zscore(s):
    return (df['baz'] - np.mean(s)) / np.std(df['baz'])

df[zscore(df['baz']).abs().gt(1)]

Result:

     foo       baz
7  43559 -27315.10
8  43592 -30326.58

That's looking for items that have a z-score of 1 or greater. If, for instance, you want to see which ones have a z-score of 2 or more:

df[zscore(df['baz']).abs().gt(2)]

Result:

     foo       baz
8  43592 -30326.58

This will show you z-scores (I'm pretty sure there are many ways to do this kind of thing).

cols = list(df.columns)
cols.remove('foo')
df[cols]

for col in cols:
    col_zscore = col + '_zscore'
    df[col_zscore] = (df[col] - df[col].mean())/df[col].std(ddof=0)
df

Take a look at this as well.

Ways to Detect and Remove the Outliers | Towards Data Science