Multivariate Time Series Anomalous Entry Detection

Question

I have a multivariate data set of the following structure.

It is a time series sequence of logs with additional string attribute columns id1 and id2. If too many entries come in a sequence that have similar values for either id1 or id2, I want to classify them as anomalies and flag them. I tried using LSTM, but I'm confused in how to use it for textual data, specially when the values that can come in as inputs are not always the same (one time window might have more entries than other).

The similar entries may not always be together but may be very frequent in a specific time window.

I am going to be classifying anomalies at the end of every day in a batch, so I'd like to feed the trained model a bunch of log entries and then find anomalies in it.

I'd appreciate any help w.r.t. to what direction to look in and if LSTM is even required for this.

Answer 1

If too many entries come in a sequence that have similar values for either id1 or id2, I want to classify them as anomalies and flag them (...) I'd appreciate any help w.r.t. to what direction to look in

General Approach

Here's how I would approach this:

1. Select the time windows, e.g. using pandas.rolling
2. For each time window, count the number of occurances for each string (per id1, id2)
3. Compare occurances to usual occurances. Usual here could be in comparision to say the moving average of occurances of the same string in the same window, across many windows, in comparison to the same window/range on the previous day etc.
4. Identify the anomalies. Anomalies would be those values that occur less or more frequently as "usual" by some threshold (say occurances within 2 standard deviations of the mean is normal, anything else is an anomaly)

A more elaborate and extensive treatment of possible approaches is given in Detecting Anomalies in System log files by Tim Zwietasch.

What algorithm to use?

This can be done with or without a machine learning algorithm:

• LSTM might be an option if the process generating these logs can be modelled as some distribution so that an LSTM is able to generate the most likely sequences. In this case an anomaly would be a sequence that has a low probability of being generated by the model.
• A simpler ML option would seem to use a "classic" anomaly detection algorithm such as one-class SVM, KNN, K-Means or LOF. A good overview is given in Introduction to Anomaly Detection by datascience.com (I have no affiliation)
• Build your own "classifier" using simple statistics.

Personally I would start out with building my own, using a subset of the data, just to understand the data and its patterns better. Then, perhaps, use machine learning for robustness and scalability.

Encoding the log files

Whatever your choice, the key will be in encoding the data. Any algorithm will model the distribution of your data in one way or another (i.e. occurances of values per time window), so you have to present the data in a way suitable to detection of anomalies as per your criteria (anomaly relative to what?).

One encoding might be as follows:

• for every time window, build one row of input data
• for every value, have one column as the number of occurances of this value
• add more columns for any other factor that could be influecing what is "normal" e.g. hour of day, day of week, week of year

Dealing with similar values

Note I did not yet treat "similar values" but assumed that the occurance counts are for exact values. However it should be relatively easy to extend the above to similar values by creating columns of occurance patterns as opposed to specific values. A pattern in this case could be e.g. all strings that contain some substring, say 'aaa' - or you could use a machine learning model to create clusters of value. Then use counts by patterns/clusters as the input instead of counts per value.

Answer 2

First step, embed the text values of id1,id2 into a vector of real-numbers. Assuming that you use Keras for the overall implementation, this can be done with the Embedding Layer.

Then, zero-pad short words or truncate long ones so that in the end you have words of the same length. These words should be vectors of real numbers and their length should be fixed.

Lastly, train your LSTM with those real-valued vectors and their labels and then test to see how well it can identify the label itself.

A very good tutorial of text binary classification with LSTMs can be found here.

Let me know what you think :)

Answer 3

I am not sure you really need any machine learning algorithm here. You could find your anomalies in linear time with one for-loop.

Answer 4

A very basic solution in python without machine learning:

import pandas as pd
testFrame = pd.DataFrame({'id1':['dsads', 'bcdaa', 'bcdaa'],'id2':['daddg','kjfjd','abcde']})
testFrame['id1_shift'] = testFrame['id1'].shift(1)
testFrame['flag'] = testFrame['id1'] == testFrame['id1_shift']

With the value for shift you can modify how many steps before and after each window you want to look. This, of course, does not account for similar sequences but only for identical ones.

If you want to use ML because the method above becomes very tedious (I could not modify it right away for similiar sequences, though that should be possible), you may want to do something like a one-hot encoder on id1 and id2, then change your dataframe to id1, id1_previous, id1_pre_previous until the memory you want to explore is reached, and then train an algorithm on labeled data.