I'm facing a problem that seems 'easy,' but I've been struggling with it for a while now in the field of anomaly/outlier detection.
I have a dataset of around 60K data points. Each data point is part of a group (~1500 groups; min group size is 15) and has a length parameter.
The task is to perform anomaly detection within the groups, i.e., if a data point has an irregular length compared to the other data points in the group, mark it as an anomaly.
I'm using a log2 transformation on the data, and after the transformation, the majority of groups (75%) are normally distributed based on Shapiro-Wilks test.
As a first solution, I tried the classical distance in std from the mean, where if the length is bigger than mean+3*std, then this is an anomaly.
I had 2 problems with this solution:
In groups with a high number of data points, where the vast majority of data points had the same or very similar length, the std was very small, thus making the threshold very small, and it resulted in alerting on data points, which I do not consider as anomalies.
This method resulted in a relatively high detection (~250 anomalies), and I aim to alert only a small number of the most extreme anomalies in my data across all the groups.
When I tried to increase the threshold, e.g., to 4std, I faced another problem, where I missed anomalies in groups where one data point had a very large length compared to the others, which resulted with a high std, and thus making the extreme data point to have a 'low' std from the mean distance.
I'd appreciate any help or thoughts on the subject. Thanks!
