I've been looking for methods that can help figure out anomalies in textual data stored in databases. Major goal is to use a unsupervised learning method to detect the anomalies. Further how can I find a context in the data set and figure out contextual anomalies?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ Can you provide a concrete example of what you might use this for? I may have been looking into a relevant technique recently (LSTM autoencoders), but the question could possibly use a little elaboration. $\endgroup$– R HillCommented Nov 7, 2016 at 12:17
-
$\begingroup$ I may use it to flag anomalies from census data or user polls where users might give erroneous and irrelevant entries to forms. Sensor data can also be covered as sensors record data automatically and something goes missing. $\endgroup$– Sanjeev RathorCommented Nov 7, 2016 at 16:15
-
$\begingroup$ Have you looked at this paper? $\endgroup$– Has QUIT--Anony-MousseCommented Nov 8, 2016 at 6:30
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
This assumes you’re happy with neural networks. If you’re not, this answer probably isn’t of much use to you.
Firstly, a little bit about anomaly detection via autoencoders. Apologies if you’re already familiar with this.
An autoencoder is a neural network which learns to reproduce its own input when compressed through a “bottleneck” layer. For example, you may want to find a lower-dimensional feature representation of a set of 100 x 100 images. Your neural network architecture has an input layer of 10,000 elements, an output layer of 10,000 elements, and one of its hidden layers will be relatively narrow compared to the input space: say 100 nodes.
The objective is to train the network to produce an output as close to the input as possible, while throwing away all but 100 nodes’ worth of activation. You are trying to produce as lossless a compression of the input as possible, so those 100 nodes should be a very information-rich representation of the kind of data you trained it on.
“How does this have any bearing on anomaly detection?” I hear you ask. Well, if you train your autoencoder on non-anomalous data, it will learn a non-anomalous lower-dimensional feature representation. This will mean the reconstruction error from pushing something through the autoencoder will be lower for data similar to what it was trained on than it will for other arbitrary data. If it receives input that is substantively different from what it was trained on, the reconstruction error will be higher. So, given a set of novel inputs, those inputs with the highest reconstruction error are the most anomalous, as they are poorly reconstructed from the non-anomalous feature representation.
If your data has a temporal structure, and you have plenty of training samples, you might want to consider constructing an autoencoding LSTM. An LSTM is a neural network architecture for encoding and decoding sequentially dependent data, and a full description of how this works is beyond both the scope of this post and my own abilities. There are many magnificent resources available online for getting to grips with this.
Here is a relevant paper on using LSTMs for anomaly detection in time series in general. It may be that LSTMs are unnecessary for your purposes if the data isn’t strongly sequentially dependent.
-
$\begingroup$ Training the module with non-anomalous training data is a fairly complex task, given that the data is temporal size of training set will expand exponentially. Covering all possible non-anomalous data is NP-H problem. Further, the solution seems promising for point anomalies but then will have high error rate with contextual anomalies. $\endgroup$ Commented Nov 7, 2016 at 17:36
-
$\begingroup$ I would like your suggestion whether NLP combined with KSOM can be used? $\endgroup$ Commented Nov 7, 2016 at 17:38
$\begingroup$
$\endgroup$
This is the first hit on google for "anomaly detection text".
It's a little old but it's likely a good starting point. It seems the author uses heuristic algorithms (several measures of distance) to define an anomaly.
Another option may be to look at the moving average of a given token (word) in your corpora over time and see if its t-stat is greater than some threshold. There are probably many, many better ways to do it but that might work depending upon your goal.
answered Nov 7, 2016 at 13:37