Determining threshold for KMeans anomaly detection

Question

I'm trying to use KMeans for anomaly detection, and I know that a threshold is needed to determine the anomalies. I've seen many articles talking about how to choose K, but none talks about how to choose this threshold.

This article using Spotify data set the threshold as zmod > 3. Another article chose the threshold as points whose distance are greater than the 80th percentile (the percentile value can be changed according to the data).

I'm wondering for the Spotify case, was the threshold chosen specific only to the Spotify data, or is this a general rule for all problems? If the former, how should I go about choosing the threshold?

Answer 1

Defining the threshold in a big data set could be difficult because you may have many false positive or false negative outliers without knowing which value is the best. It depends on the data quality and sparcity: I don't think there is a simple rule.

The fastest way may be to proceed by dichotomy, potentially with a genetic algorithm.

After setting a threshold value and getting outliers, you can get a sample if there are many and see if they have been classified correctly. Then repeat by increasing or decreasing the threshold value until reaching a satisfactory result.