1
$\begingroup$

I'm trying to use KMeans for anomaly detection, and I know that a threshold is needed to determine the anomalies. I've seen many articles talking about how to choose K, but none talks about how to choose this threshold.

This article using Spotify data set the threshold as zmod > 3. Another article chose the threshold as points whose distance are greater than the 80th percentile (the percentile value can be changed according to the data).

I'm wondering for the Spotify case, was the threshold chosen specific only to the Spotify data, or is this a general rule for all problems? If the former, how should I go about choosing the threshold?

$\endgroup$

1 Answer 1

1
$\begingroup$

Defining the threshold in a big data set could be difficult because you may have many false positive or false negative outliers without knowing which value is the best. It depends on the data quality and sparcity: I don't think there is a simple rule.

The fastest way may be to proceed by dichotomy, potentially with a genetic algorithm.

After setting a threshold value and getting outliers, you can get a sample if there are many and see if they have been classified correctly. Then repeat by increasing or decreasing the threshold value until reaching a satisfactory result.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.