# Anomaly Detection/Novelty detection

I have a data-set that has over 6 million normal data and around 50 anomaly data. Those anomaly data is identified manually (by monitoring the user`s activity over camera and identify). I need to develop a model to detect these anomalies.

My problem is that the anomaly data looks like normal data, which means they are not outliers or has a certain pattern. If I plot the normal data over anomaly data they are in the same distribution.

I tried several anomaly detection approaches:

1. Multivariate Gaussian Distribution Approach to identify anomalies

• I tried to create new features that anomaly data will be outliers and then I can use Multivariate Gaussian Distribution Approach, but could not able to find any combination to isolate the anomalies.
2. I guess there is no point of using a classification algorithm since dataset is highly imbalanced.

• I tried OneClassSVM, DecisionTree, RandomForest but AUC is 0.5 (as good as random).

How to implement a model for this kind of scenario?

Other methods which I can think about:

• Develop a NN with AutoEncoders
• Try Generate Synthetic Samples and resample the dataset
• How many features does your data have? How about using Lasso Regression to select features (using 50 anomaly data and another random 50/100 normal data, assuming that normal data come from the same distribution) and then see when plotting normal vs. abnormal data points, are abnormal points separated out? Feb 10 '20 at 7:43
• How can you tell they are anomalous data? If they are not outliers, statistically speaking, where can you tell they are anomalies? There must be some difference that lets your mind separate them from the rest. Let's follow this hint to find the right features for your model Mar 11 '20 at 8:04
• that specific records are identified as anomalous by looking at the user behaviour at that specific time..Issue is they are not outliers,they are exactly looks like normal data. Mar 11 '20 at 10:37
• ok, but how can you as a human say it's an anomaly? In plain words, not mathematically. Mar 11 '20 at 10:43
• dataset is related daily usage of pachinko and slot (pinball ) game users.for example customer has identified some particular machine,at specific time some user has done a fraud or illegal game play(monitoring user behaviors).so he recorded that entry as a fraudulent or illegal input to the machine at that specific time.dataset is already separated to normal and anomaly data. Mar 11 '20 at 10:56