I have a large multi dimensional dataset that is generated each day.
What would be a good approach to detect any kind of 'anomaly' as compared with previous days? Is this a suitable problem that could be addressed with neural networks?
Any suggestions are appreciated.
Additional information: there are no examples, so the method should detect the anomalies itself
From the formulation of the question, I assume that there are no "examples" of anomalies (i.e. labels) whatsoever. With that assumption, a feasible approach would be to use autoencoders: neural networks that receive as input your data and are trained to output that very same data. The idea is that the training has allowed the net to learn representations of the input data distributions in the form of latent variables.
There is a type of autoencoder called denoising autoencoder, which is trained with corrupted versions of the original data as input and with the uncorrupted original data as output. This delivers a network that can remove noise (i.e. data corruptions) from the inputs.
You may train a denoising autoencoder with the daily data. Then use it on new daily data; this way you have the original daily data and an uncorrupted version of those very same data. You can then compare both to detect significant differences.
The key here is which definition of significant difference you choose. You could compute the euclidean distance and assume that if it surpasses certain arbitrary threshold, you have an anomaly. Another important factor is the kind of corruptions you introduce; they should be as close as possible to reasonable abnormalities.
Another option would be to use Generative Adversarial Networks. The byproduct of the training is a discriminator network that tells apart normal daily data from abnormal data.
I think that heavily depends on the nature of your data (categorical/continuous). I'd start with simple methods first. Those come to my mind:
If none of these are suitable, then there is whole branch of stats/ML models specialized for anomaly detection. SVM, t-SNE, Isolation forests, Peer Group Analysis, Break Point Analysis, Time series (where you would look for outliers outside trends).
Those methods have the advantage that they are sort of white-box, so you can tell why someone is an outlier. Should this not be the thing you want, others suggested ANN approaches, which will also work.
I am trying to solve a similar problem. Does your dataset contain a mix of text and numerical features ? If so the complexity of detecting anomalies increases ( I don't know by what factor). If your dataset is uniform, for example containing only numeric values, you can potentially use a RNN which still needs a labelled dataset but it can detect time series like patterns ( since you mention comparison with pervious day's values for ex)
A simple way to do this using Autoencoders (without "denoising autoencoders" that need to be trained with "corrupted data") is to train an autoencoder and then inspect the RMSE of the rows from the input that didn't decode well (the ones that the autoencoder had a hard time reconstructing). By some definition that data would represent an anomaly (certainly this would be the case for things like spikes in traffic).
