I have a large multi dimensional dataset that is generated each day.

What would be a good approach to detect any kind of 'anomaly' as compared with previous days? Is this a suitable problem that could be addressed with neural networks?

Any suggestions are appreciated.

additional information: there are no examples, so the method should detect the anomalies itself

up vote 10 down vote accepted

From the formulation of the question, I assume that there are no "examples" of anomalies (i.e. labels) whatsoever. With that assumption, a feasible approach would be to use autoencoders: neural networks that receive as input your data and are trained to output that very same data. The idea is that the training has allowed the net to learn representations of the input data distributions in the form of latent variables.

There is a type of autoencoder called denoising autoencoder, which is trained with corrupted versions of the original data as input and with the uncorrupted original data as output. This delivers a network that can remove noise (i.e. data corruptions) from the inputs.

You may train a denoising autoencoder with the daily data. Then use it on new daily data; this way you have the original daily data and an uncorrupted version of those very same data. You can then compare both to detect significant differences.

The key here is which definition of significant difference you choose. You could compute the euclidean distance and assume that if it surpasses certain arbitrary threshold, you have an anomaly. Another important factor is the kind of corruptions you introduce; they should be as close as possible to reasonable abnormalities.

Another option would be to use Generative Adversarial Networks. The byproduct of the training is a discriminator network that tells apart normal daily data from abnormal data.

I think that heavily depends on the nature of your data (categorical/continuous). I'd start with simple methods first. Those come to my mind:

  • You can compare distribution of each variable either by using quantiles or any statistical test to see whether they are significantly different
  • You could also count occurrence of each label/category and compare them
  • I'd also try to employ any sort of distance measure. For example you could calculate mahalanobis distance and look for big changes
  • Or something really simple - just an absolute difference between new and old data, set a threshold and everything exceeding the threshold will be reported
  • You can also put in place some multidimensional techniques - like correlation matrix, principal components, clustering etc. and look for changes

If none of these are suitable, then there is whole branch of stats/ML models specialized for anomaly detection. SVM, t-SNE, Isolation forests, Peer Group Analysis, Break Point Analysis, Time series (where you would look for outliers outside trends).

Those methods have the advantage that they are sort of white-box, so you can tell why someone is an outlier. Should this not be the thing you want, others suggested ANN approaches, which will also work.

I am trying to solve a similar problem. Does your dataset contain a mix of text and numerical features ? If so the complexity of detecting anomalies increases ( I don't know by what factor). If your dataset is uniform, for example containing only numeric values, you can potentially use a RNN which still needs a labelled dataset but it can detect time series like patterns ( since you mention comparison with pervious day's values for ex)

A simple way to do this using Autoencoders (without "denoising autoencoders" that need to be trained with "corrupted data") is to train an autoencoder and then inspect the RMSE of the rows from the input that didn't decode well (the ones that the autoencoder had a hard time reconstructing). By some definition that data would represent an anomaly (certainly this would be the case for things like spikes in traffic).

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.