Problem Overview: I am currently working on a project involving anomaly detection in log data. The anomalies are defined by deviations from historical patterns. The log data has a simple structure: [timestamp: log_statement].

Dataset Details: The dataset consists of logs in the format [timestamp: log_statement] and have 10k+ logs Each log_statement has been processed to generate keys (k1, k2, ...) for uniqueness after that it became 200-250 unique keys.

Current Approach: I have preprocessed the log data to extract log statements and assigned unique keys to them. Utilized LSTM for training with a window-based approach, considering past logs for predictions (similar to next word prediction by looking past words). Anomalies are detected by comparing predicted and actual log statements within the specified window.

Specific Questions:

  1. Is the current approach suitable for capturing temporal patterns in log data, am I right converting each log_statement to keys: k1,k2..etc ?
  2. How can I ensure that the LSTM model is effectively learning and representing the patterns?
  3. Are there alternative methods, either statistical, machine learning-based, deep-learning based, that might offer better results and more interpretable?
  4. Also are there any ways, where I can tell after training that I am confident about these keys(say k1,k2) these are predictable and rest are not. I thought to compare True positive and True negative of each key, but it will heavily dependent on the model.

1 Answer 1


The anomalies are defined by deviations from historical patterns.

I respectfully disagree. I mean, sure, we identify anomalies by observing deviations from historical patterns. But we're doing AD here, and I feel definitions matter.

In most modeling efforts we believe there is a Generating Process out in the world, it yields observables, and our model can learn the structure and predict observations. (The generating process could be compound, as with MoG.)

In Anomaly Detection we believe there is a Normal generating process producing most observed examples, and an Anomalous (abnormal) generating process that generates a minority of observations. Examples include a formerly loyal employee who suddenly starts responding to incentives from a competitor firm, server access from clients that might be operated by trusted staff or by a criminal, and performance measurements of equipment that might suffer from wear or failure of a component. In each case the nature of one generating process is different, and we believe an AD classification model can infer such a difference by examining the log of what was observed.

business domain

Each log_statement has been processed to generate keys (k1, k2, ...)

You haven't introduced the domain this logged data is drawn from, and I fear the unique keys might be insufficient.

Suppose that logged examples come from card swipes on a building's door reader, initially looking like

(10:15, Alice, entering)
(17:01, Alice, exiting)

It sounds like your anonymizing data collection protocol would map "Alice entering" to k1, and "Alice exiting" to k2, thereby obscuring the fact that the keys are related. We might want to model things like elapsed time Alice spent in the office building, or a prohibition on sleeping overnight there.

Consider reviewing how you anonymize. Even if you can't tell that k1 corresponds to a particular human, Alice, you may want to know that k1 describes the same human that k2 describes. If there were causal relationships in the original data, you want the anonymized dataset to still reveal them.


I didn't hear you describe any Ground Truth training labels. That can make it hard to ensure your LSTM model is effectively learning and representing the patterns, if you don't have a basis for evaluation. An approach to AD that people sometimes take is training a model to predict future events very well, and then any mis-predictions are labeled anomalous. If you come at it from the alternative perspective that there are two kinds of generating process in the world, and the model is responsible for labeling which process yielded an observation, then we stop rewarding the model for doing a crummy job of predicting. We will need some labels, though.

There are many alternative ML and statistical approaches you could use other than LSTM. The OP doesn't reveal enough about the underlying problem to be able to say much concretely. For example, if observations were card swipes at a door, and business is ramping up or exhibits seasonal effects, we might need a transform that will feed de-trended stationary data to our primary model. I will mention some very simple transforms.

Often a dataset reveals human behavior; we know things about such behavior, and can add that knowledge to the dataset. Augmenting logged events with synthetic events like local sunrise and sunset can be helpful. Or 9am and 5pm if regular work hours are relevant, plus perhaps times describing normal start and end of when folks eat lunch.

Based on time alone you may know the model has little predictive power, perhaps at midnight or on a weekend, due to small number of examples generated then. Consider discarding logged entries from such intervals.

Simple counts over recurring intervals can be a surprisingly powerful feature to extract. Daily counts might be noisy, while a 7- or 28- day aggregate smooths that out, and automatically deals with seasonality effects. Resist the urge to produce monthly aggregates, as then you may be faced with counting how many weekdays are in one interval versus another.

Predicting rank order can be much easier than predicting an absolute count. Suppose that k3 and k5 correspond to different people working in the same group, and one person is more experienced, more productive. As the group takes on more work, or schedules a holiday, we might see event counts on those keys rise or fall in tandem, while still roughly maintaining their rank ordering.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.