I have a multivariate data set of the following structure. enter image description here

It is a time series sequence of logs with additional string attribute columns id1 and id2. If too many entries come in a sequence that have similar values for either id1 or id2, I want to classify them as anomalies and flag them. I tried using LSTM, but I'm confused in how to use it for textual data, specially when the values that can come in as inputs are not always the same (one time window might have more entries than other).

The similar entries may not always be together but may be very frequent in a specific time window.

I am going to be classifying anomalies at the end of every day in a batch, so I'd like to feed the trained model a bunch of log entries and then find anomalies in it.

I'd appreciate any help w.r.t. to what direction to look in and if LSTM is even required for this.


4 Answers 4


If too many entries come in a sequence that have similar values for either id1 or id2, I want to classify them as anomalies and flag them (...) I'd appreciate any help w.r.t. to what direction to look in

General Approach

Here's how I would approach this:

  1. Select the time windows, e.g. using pandas.rolling
  2. For each time window, count the number of occurances for each string (per id1, id2)
  3. Compare occurances to usual occurances. Usual here could be in comparision to say the moving average of occurances of the same string in the same window, across many windows, in comparison to the same window/range on the previous day etc.
  4. Identify the anomalies. Anomalies would be those values that occur less or more frequently as "usual" by some threshold (say occurances within 2 standard deviations of the mean is normal, anything else is an anomaly)

A more elaborate and extensive treatment of possible approaches is given in Detecting Anomalies in System log files by Tim Zwietasch.

What algorithm to use?

This can be done with or without a machine learning algorithm:

  • LSTM might be an option if the process generating these logs can be modelled as some distribution so that an LSTM is able to generate the most likely sequences. In this case an anomaly would be a sequence that has a low probability of being generated by the model.
  • A simpler ML option would seem to use a "classic" anomaly detection algorithm such as one-class SVM, KNN, K-Means or LOF. A good overview is given in Introduction to Anomaly Detection by datascience.com (I have no affiliation)
  • Build your own "classifier" using simple statistics.

Personally I would start out with building my own, using a subset of the data, just to understand the data and its patterns better. Then, perhaps, use machine learning for robustness and scalability.

Encoding the log files

Whatever your choice, the key will be in encoding the data. Any algorithm will model the distribution of your data in one way or another (i.e. occurances of values per time window), so you have to present the data in a way suitable to detection of anomalies as per your criteria (anomaly relative to what?).

One encoding might be as follows:

  • for every time window, build one row of input data
  • for every value, have one column as the number of occurances of this value
  • add more columns for any other factor that could be influecing what is "normal" e.g. hour of day, day of week, week of year

Dealing with similar values

Note I did not yet treat "similar values" but assumed that the occurance counts are for exact values. However it should be relatively easy to extend the above to similar values by creating columns of occurance patterns as opposed to specific values. A pattern in this case could be e.g. all strings that contain some substring, say 'aaa' - or you could use a machine learning model to create clusters of value. Then use counts by patterns/clusters as the input instead of counts per value.

  • 1
    $\begingroup$ Thanks, yes I ended up writing a solution quite similar to your first approach. I wrote a custom churn model, where I calculated averages over multiple time windows and used an XGboost classifier. Although KNN worked pretty good too, but XGboost gave a much better recall score which was something crucial to my project. $\endgroup$ Commented Jan 16, 2019 at 7:58

First step, embed the text values of id1,id2 into a vector of real-numbers. Assuming that you use Keras for the overall implementation, this can be done with the Embedding Layer.

Then, zero-pad short words or truncate long ones so that in the end you have words of the same length. These words should be vectors of real numbers and their length should be fixed.

Lastly, train your LSTM with those real-valued vectors and their labels and then test to see how well it can identify the label itself.

A very good tutorial of text binary classification with LSTMs can be found here.

Let me know what you think :)

  • $\begingroup$ Thanks for the answer. I just have a query here. The words id1 and id2 are not always going to be the same length. I can zero pad them, but the catch here would be that the next word that comes in, might be of a length that the model has not seen yet and truncating it can make it similar to another word that the system has already seen and is different than the current one. $\endgroup$ Commented Jun 13, 2018 at 17:46
  • $\begingroup$ I see. I would expect that the training dataset would be representative of the length of the words that you would expect to see during testing, therefore you will know "how long" the padding should be. Since your problem has only 2 inputs, you can afford (computationally) a conservatively long padded word, such as +2/3/4/5 characters to the maximum word length that was observed during training. Meaning that if the longest word during training is 6 chars, pad all words to 10 chars and concatenate longer ones down to 10. It is definitely worth trying :) $\endgroup$
    – pcko1
    Commented Jun 13, 2018 at 17:55
  • $\begingroup$ Okay, I'll try with a large enough padding. Now once I have the embedded data ready. I'm still not sure how I'm going to pass the data as sequence and most importantly how can i test (predict) out chunks of data every day and flag just the entries which follow a pattern and are anomalous? $\endgroup$ Commented Jun 13, 2018 at 18:07
  • $\begingroup$ I think this post's problem is described as anomaly detection in text, which means sequence classification. This is excellenty explained in the link I provided you with in my answer. If you want to predict future values, that's another story. You should look for sequence2sequence LSTM for that purpose. A helpful link for that is: machinelearningmastery.com/…. $\endgroup$
    – pcko1
    Commented Jun 13, 2018 at 18:22

I am not sure you really need any machine learning algorithm here. You could find your anomalies in linear time with one for-loop.

  • $\begingroup$ Can you please elaborate on that? $\endgroup$ Commented Jul 13, 2018 at 20:48

A very basic solution in python without machine learning:

import pandas as pd
testFrame = pd.DataFrame({'id1':['dsads', 'bcdaa', 'bcdaa'],'id2':['daddg','kjfjd','abcde']})
testFrame['id1_shift'] = testFrame['id1'].shift(1)
testFrame['flag'] = testFrame['id1'] == testFrame['id1_shift']

With the value for shift you can modify how many steps before and after each window you want to look. This, of course, does not account for similar sequences but only for identical ones.

If you want to use ML because the method above becomes very tedious (I could not modify it right away for similiar sequences, though that should be possible), you may want to do something like a one-hot encoder on id1 and id2, then change your dataframe to id1, id1_previous, id1_pre_previous until the memory you want to explore is reached, and then train an algorithm on labeled data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.