I have a data-set that has over 6 million normal data and around 50 anomaly data. Those anomaly data is identified manually (by monitoring the user`s activity over camera and identify). I need to develop a model to detect these anomalies.
My problem is that the anomaly data looks like normal data, which means they are not outliers or has a certain pattern. If I plot the normal data over anomaly data they are in the same distribution.
I tried several anomaly detection approaches:
Multivariate Gaussian Distribution Approach to identify anomalies
- I tried to create new features that anomaly data will be outliers and then I can use Multivariate Gaussian Distribution Approach, but could not able to find any combination to isolate the anomalies.
I guess there is no point of using a classification algorithm since dataset is highly imbalanced.
- I tried OneClassSVM, DecisionTree, RandomForest but AUC is 0.5 (as good as random).
How to implement a model for this kind of scenario?
Other methods which I can think about:
- Develop a NN with AutoEncoders
- Try Generate Synthetic Samples and resample the dataset