If too many entries come in a sequence that have similar values for either id1 or id2, I want to classify them as anomalies and flag them (...) I'd appreciate any help w.r.t. to what direction to look in
Here's how I would approach this:
- Select the time windows, e.g. using
- For each time window, count the number of occurances for each string (per id1, id2)
- Compare occurances to usual occurances. Usual here could be in comparision to say the moving average of occurances of the same string in the same window, across many windows, in comparison to the same window/range on the previous day etc.
- Identify the anomalies. Anomalies would be those values that occur less or more frequently as "usual" by some threshold (say occurances within 2 standard deviations of the mean is normal, anything else is an anomaly)
A more elaborate and extensive treatment of possible approaches is given in Detecting Anomalies in System log files by Tim Zwietasch.
What algorithm to use?
This can be done with or without a machine learning algorithm:
- LSTM might be an option if the process generating these logs can be modelled as some distribution so that an LSTM is able to generate the most likely sequences. In this case an anomaly would be a sequence that has a low probability of being generated by the model.
- A simpler ML option would seem to use a "classic" anomaly detection algorithm such as one-class SVM, KNN, K-Means or LOF. A good overview is given in Introduction to Anomaly Detection by datascience.com (I have no affiliation)
- Build your own "classifier" using simple statistics.
Personally I would start out with building my own, using a subset of the data, just to understand the data and its patterns better. Then, perhaps, use machine learning for robustness and scalability.
Encoding the log files
Whatever your choice, the key will be in encoding the data. Any algorithm will model the distribution of your data in one way or another (i.e. occurances of values per time window), so you have to present the data in a way suitable to detection of anomalies as per your criteria (anomaly relative to what?).
One encoding might be as follows:
- for every time window, build one row of input data
- for every value, have one column as the number of occurances of this value
- add more columns for any other factor that could be influecing what is "normal" e.g. hour of day, day of week, week of year
Dealing with similar values
Note I did not yet treat "similar values" but assumed that the occurance counts are for exact values. However it should be relatively easy to extend the above to similar values by creating columns of occurance patterns as opposed to specific values. A pattern in this case could be e.g. all strings that contain some substring, say 'aaa' - or you could use a machine learning model to create clusters of value. Then use counts by patterns/clusters as the input instead of counts per value.