2
$\begingroup$

I am performing anomaly detection on different datasets and thought to first cluster the dataset and submit each of the clusters to different AD models. I am using HDBSCAN, and in my test dataset I get anywhere between 10 and 20 clusters, but when I ran the first test in production I get 3500. How can I repeat the AD models dynamically amongst all the clusters?

$\endgroup$
  • $\begingroup$ Avoid clustering, esp. as a means to achieve another goal. Try learning a more stable representation; look into embeddings. How big are your training and production datasets? $\endgroup$ – Emre May 21 '18 at 18:48
  • $\begingroup$ Thanks, but I think that clustering adds a means to study anomalies according to behavior. I don't want to tinker with the data, just create an AD framework. But regardless of this, the question remains the same if I would want to regress or perform any other sort of analysis on a per cluster basis. I guess I could force it by creating partitions, but this arrests the scope of the exercise. $\endgroup$ – Lou65 May 21 '18 at 19:25
0
$\begingroup$

Considering your objective, i would suggest you to use LOF ( Local Outlier Factor) based clustering. This will give you outliers respective of clusters, Not only the the global outliers. LOF distance of all the data points would be used to identify abnormalities. Here you dont have to be worry about number of clusters.

https://en.wikipedia.org/wiki/Local_outlier_factor

Also i would question the need of having multiple clustering algos. They are intended to use for specfic scenarios. One should see the underlying distribution and pick the best AD algo.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.