1
$\begingroup$

I have a data-set that has over 6 million normal data and around 50 anomaly data. Those anomaly data is identified manually (by monitoring the user`s activity over camera and identify). I need to develop a model to detect these anomalies.

My problem is that the anomaly data looks like normal data, which means they are not outliers or has a certain pattern. If I plot the normal data over anomaly data they are in the same distribution.

I tried several anomaly detection approaches:

  1. Multivariate Gaussian Distribution Approach to identify anomalies

    • I tried to create new features that anomaly data will be outliers and then I can use Multivariate Gaussian Distribution Approach, but could not able to find any combination to isolate the anomalies.

  2. I guess there is no point of using a classification algorithm since dataset is highly imbalanced.

    • I tried OneClassSVM, DecisionTree, RandomForest but AUC is 0.5 (as good as random).

How to implement a model for this kind of scenario?

Other methods which I can think about:

  • Develop a NN with AutoEncoders
  • Try Generate Synthetic Samples and resample the dataset
Improve this question
$\endgroup$
7
  • $\begingroup$ How many features does your data have? How about using Lasso Regression to select features (using 50 anomaly data and another random 50/100 normal data, assuming that normal data come from the same distribution) and then see when plotting normal vs. abnormal data points, are abnormal points separated out? $\endgroup$
    – Duc Nguyen
    Feb 10, 2020 at 7:43
  • $\begingroup$ How can you tell they are anomalous data? If they are not outliers, statistically speaking, where can you tell they are anomalies? There must be some difference that lets your mind separate them from the rest. Let's follow this hint to find the right features for your model $\endgroup$
    – Leevo
    Mar 11, 2020 at 8:04
  • $\begingroup$ that specific records are identified as anomalous by looking at the user behaviour at that specific time..Issue is they are not outliers,they are exactly looks like normal data. $\endgroup$
    – Rajith Thennakoon
    Mar 11, 2020 at 10:37
  • $\begingroup$ ok, but how can you as a human say it's an anomaly? In plain words, not mathematically. $\endgroup$
    – Leevo
    Mar 11, 2020 at 10:43
  • $\begingroup$ dataset is related daily usage of pachinko and slot (pinball ) game users.for example customer has identified some particular machine,at specific time some user has done a fraud or illegal game play(monitoring user behaviors).so he recorded that entry as a fraudulent or illegal input to the machine at that specific time.dataset is already separated to normal and anomaly data. $\endgroup$
    – Rajith Thennakoon
    Mar 11, 2020 at 10:56

1 Answer 1

Reset to default
1
$\begingroup$

I believe you can use a classification algorithm where you manually overrepresent the "anomalies" class. By how much, depends on the cost induced by the anomalies.

Just to illustrate what I mean:

Anomalies cover a continuum between two extremes: 1) those which can be safely ignored, because they induce no costs, and 2) those whose costs is unbearable. As an example close to the extreme (1), imagine a flower shop getting an order for Iris virginica. The customer will hardly object if, among 6 million flowers shipped, 50 actually belong to the species Iris versicolor.

On the other hand, if you are shipping airplane parts, and a faulty part would cause a disaster with 300+ dead, then you are closer to the extreme (2). You will likely want to avoid shipping even one faulty part.

In practice, it will have to be a trade-off: You can be on the safe side if you ship no parts at all, but then you go bankrupt. On the other hand, if you do no quality assurance at all, you ship enough faulty parts to cause a disaster in a foreseeable future, have to pay damages, and go bankrupt again.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.