The anomalies are defined by deviations from historical patterns.
I respectfully disagree.
I mean, sure, we identify anomalies by observing deviations from historical patterns.
But we're doing AD here, and I feel definitions matter.
In most modeling efforts we believe there is a Generating Process out in the world,
it yields observables, and our model can learn the structure and predict
observations.
(The generating process could be compound, as with
MoG.)
In Anomaly Detection we believe there is a Normal generating process
producing most observed examples, and an Anomalous (abnormal) generating process
that generates a minority of observations.
Examples include a formerly loyal employee who suddenly starts responding
to incentives from a competitor firm,
server access from clients that might be operated by trusted staff or
by a criminal,
and performance measurements of equipment that might suffer from wear
or failure of a component.
In each case the nature of one generating process is different,
and we believe an AD classification model can infer such a difference
by examining the log of what was observed.
business domain
Each log_statement has been processed to generate keys (k1, k2, ...)
You haven't introduced the domain this logged data is drawn from,
and I fear the unique keys might be insufficient.
Suppose that logged examples come from card swipes on a building's door reader,
initially looking like
(10:15, Alice, entering)
(17:01, Alice, exiting)
It sounds like your anonymizing data collection protocol would
map "Alice entering" to k1, and "Alice exiting" to k2,
thereby obscuring the fact that the keys are related.
We might want to model things like elapsed time Alice spent
in the office building, or a prohibition on sleeping overnight there.
Consider reviewing how you anonymize.
Even if you can't tell that k1 corresponds to a particular human, Alice,
you may want to know that k1 describes the same human that k2 describes.
If there were causal relationships in the original data,
you want the anonymized dataset to still reveal them.
unsupervised
I didn't hear you describe any Ground Truth training labels.
That can make it hard to ensure your LSTM model is effectively learning
and representing the patterns, if you don't have a basis for evaluation.
An approach to AD that people sometimes take is training a model to
predict future events very well, and then any mis-predictions are
labeled anomalous.
If you come at it from the alternative perspective that there are two
kinds of generating process in the world, and the model is responsible
for labeling which process yielded an observation, then we stop
rewarding the model for doing a crummy job of predicting.
We will need some labels, though.
There are many alternative ML and statistical approaches you could use
other than LSTM.
The OP doesn't reveal enough about the underlying problem
to be able to say much concretely.
For example, if observations were card swipes at a door,
and business is ramping up or exhibits seasonal effects,
we might need a transform that will feed de-trended stationary data
to our primary model.
I will mention some very simple transforms.
Often a dataset reveals human behavior; we know things about such behavior,
and can add that knowledge to the dataset.
Augmenting logged events with synthetic events like local sunrise and sunset
can be helpful. Or 9am and 5pm if regular work hours are relevant,
plus perhaps times describing normal start and end of when folks eat lunch.
Based on time alone you may know the model has little predictive power,
perhaps at midnight or on a weekend, due to small number of examples
generated then.
Consider discarding logged entries from such intervals.
Simple counts over recurring intervals can be a surprisingly powerful
feature to extract. Daily counts might be noisy, while a 7- or 28- day
aggregate smooths that out, and automatically deals with seasonality effects.
Resist the urge to produce monthly aggregates, as then you may be faced
with counting how many weekdays are in one interval versus another.
Predicting rank order can be much easier than predicting an absolute count.
Suppose that k3 and k5 correspond to different people working in the same group,
and one person is more experienced, more productive.
As the group takes on more work, or schedules a holiday,
we might see event counts on those keys rise or fall in tandem,
while still roughly maintaining their rank ordering.
