I have been building a model to find explanation of Outliers in a high dimensional numerical data, generated from many sensors. The data contains more than 350 different fields and each field has numerical values (Float or Integer). It looks like: 350 columns and many rows. I want to find the outliers/ anomalies in the data and also the Explanation why those values are outliers.

I was reading about Generative models and found that "They have the potential to understand and explain the underlying structure of the Input data even when there are no labels." I would like to know if it will be good to use the GANs for outlier detection and explanation on numerical data?

  • $\begingroup$ do you solve this problem? I have the same task than you. Find outliers and describy why they are outliers. Can you guide me about methods? $\endgroup$ – WrongMath Sep 25 at 16:29
  • $\begingroup$ @WrongMath Thanks for your question, I did not use GANs for the outlier explanation in my research. I used a very basic approach inspired by Macrobase. You can go through their approach and develop the idea behind it. macrobase.stanford.edu/docs $\endgroup$ – Shashank Srivastava Sep 30 at 7:47

See this answer by Ian Goodfellow (the creator of GANs) to the same question at Quora:

There are definitely some papers about it, such as [1703.05921] Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery. I don’t work on anomaly detection so I haven’t read these papers and don’t know a lot about how they work.

It’s worth mentioning that there are also papers on anomaly detection using inputs that resemble adversarial examples: https://arxiv.org/pdf/1706.02690.pdf

For GANs, one thing to keep in mind is that the discriminator is not a generalized detector of weird things. It is trying to tell whether a sample came from the real data or one specific non-data distribution: the generator. Because of that, it seems like the discriminator would only be useful for anomaly detection if you think you can make your generator resemble the anomalies you expect to need to detect.

Nevertheless, neural networks in general are black boxes (i.e. not very interpretable), so they might detect outliers but you would not be able to tell what characterized them as so.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.