0
$\begingroup$

I'm trying to use k-means to detect anomalies in the Amount column. I have the following part of my dataset:

8   2018-09-06  -2760.14
9   2018-10-04  -1149.73
10  2018-11-07  -1551.41
11  2018-12-06  -1208.17
12  2019-01-07   -244.02
13  2019-02-06    111.00
14  2019-03-06    139.24
15  2019-04-04 -27315.10
16  2019-05-07 -30326.58
17  2019-06-06  -1633.44

I used 2 as number of clusters and I get the following centroids: [[ -237.045] [-28820.84 ]]
In my case the anomalies are rows 15 and 16 but now they form their own cluster so the distance to their centroid will be insignificant and thus not an anomaly
My question is, how can I detect the anomalies if they form their own cluster?

$\endgroup$
1
$\begingroup$

K-means is sensitive to extreme values, such as outliers.

This is because A) it assigns every point to a cluster, but outliers shouldn't be clustered, and B) it minimizes the sum-of-squares, which puts more weight on far instances such as outliers. Larger errors get even larger, and hence k-means tries to minimize the cost of outliers by making them cluster centers.

It's the wrong tool for this problem, you don't have a clustering problem.

$\endgroup$
2
$\begingroup$

I don't suggest to use k-Means clustering for outlier detection. You can check the outlierness of observations by taking the standardized distance of each observation from the series' trend.

Alternatively, you can use DBSCAN algorithms: they are clustering models specifically designed to isolate outliers. I don't know if it makes sens on time series data. I'd go for the first method.

$\endgroup$
2
  • $\begingroup$ can you please tell me why k-means is not adapted to my situation? $\endgroup$
    – mobelahcen
    Jun 18 '19 at 13:41
  • $\begingroup$ k-Means is a clustering algorithm. It is meant to divide observations in groups, to segment a dataset. You are looking to outliers, that is, observations that by definition are "not clustered" together with others. It's not a technique for outlier detection. $\endgroup$
    – Leevo
    Jun 18 '19 at 14:22
1
$\begingroup$

This seems like a z-score problem. Try this.

import pandas as pd
from scipy import stats
import numpy as np

df = pd.DataFrame({'foo': ['43349', '43377',    '43411',    '43440',    '43472',    '43502',    '43530',    '43559',    '43592',    '43622'],
                       'baz': [-2760.14,    -1149.73,   -1551.41,   -1208.17,   -244.02,    111,    139.24, -27315.1,   -30326.58,  -1633.44]})
# print(df)

def zscore(s):
    return (df['baz'] - np.mean(s)) / np.std(df['baz'])

df[zscore(df['baz']).abs().gt(1)]

Result:

     foo       baz
7  43559 -27315.10
8  43592 -30326.58

That's looking for items that have a z-score of 1 or greater. If, for instance, you want to see which ones have a z-score of 2 or more:

df[zscore(df['baz']).abs().gt(2)]

Result:

     foo       baz
8  43592 -30326.58

This will show you z-scores (I'm pretty sure there are many ways to do this kind of thing).

cols = list(df.columns)
cols.remove('foo')
df[cols]

for col in cols:
    col_zscore = col + '_zscore'
    df[col_zscore] = (df[col] - df[col].mean())/df[col].std(ddof=0)
df

Take a look at this as well.

Ways to Detect and Remove the Outliers | Towards Data Science

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.