I am performing anomaly detection on different datasets and thought to first cluster the dataset and submit each of the clusters to different AD models. I am using HDBSCAN, and in my test dataset I get anywhere between 10 and 20 clusters, but when I ran the first test in production I get 3500. How can I repeat the AD models dynamically amongst all the clusters?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Avoid clustering, esp. as a means to achieve another goal. Try learning a more stable representation; look into embeddings. How big are your training and production datasets? $\endgroup$– EmreCommented May 21, 2018 at 18:48
-
$\begingroup$ Thanks, but I think that clustering adds a means to study anomalies according to behavior. I don't want to tinker with the data, just create an AD framework. But regardless of this, the question remains the same if I would want to regress or perform any other sort of analysis on a per cluster basis. I guess I could force it by creating partitions, but this arrests the scope of the exercise. $\endgroup$– Luis CardozoCommented May 21, 2018 at 19:25
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Considering your objective, i would suggest you to use LOF ( Local Outlier Factor) based clustering. This will give you outliers respective of clusters, Not only the the global outliers. LOF distance of all the data points would be used to identify abnormalities. Here you dont have to be worry about number of clusters.
https://en.wikipedia.org/wiki/Local_outlier_factor
Also i would question the need of having multiple clustering algos. They are intended to use for specfic scenarios. One should see the underlying distribution and pick the best AD algo.
