How would I apply anomaly detection to time series data in LSTM?

Question

I am using a LSTM RNN in Python and have successfully completed the prediction phase. My ultimate goal is anomaly detection. I'm hoping to have something like what you could see on Facebook Prophet, with anomalies marked as black dots below:

_{(source: research.fb.com)}

I've read loads of articles about how to classify with text/sequence data but there's not much on univariate time series data- only timestamps and randomly generated values with a few anomalies. One answer linked to the following method:

Anomaly detection is done by using the prediction errors as anomaly indicators.Prediction error is the difference between prediction made at time t−1 and the in-put value received at time t. The prediction errors from training data are modeled using a Gaussian distribution. The parameters of the Gaussian, mean and variance, are computed using maximum likelihood estimation (MLE). On new data, the log probability densities (PDs) of errors are calculated and used as anomaly scores: with lower values indicating a greater likelihood of the observation being an anomaly. A validation set containing both normal data and anomalies is used to set a threshold on log PD values that can separate anomalies from normal observations and incur as few false positives as possible. A separate test set is used to evaluate the model. Source

I understand overall why this method would be used but I have little idea of how to implement it with a Sequential Keras model. Does anyone have or know of some example code? Bonus if there is a way to visualize this because I don't know how I would go about that side of things either.

Please help and thank you!

Answer 1

I've been in that situation before,

there's this article on medium where the guy uses keras,tf for predicting credit card fraud detection using autoencoders which have Dense layers, but you can try the same with LSTM, can't say for sure whether it will work, but if in case it doesn't work, please try Conv1d because nowadays convolutional networks are more promising than LSTMs and GRUs -> source

Even if autoencoder doesn't help, that guy has visualizations for seeing anomalies, you must at least try them.

Hope this helps.

Answer 2

This is an old question, but I feel like a simple answer is missing, so here it goes:

1. Use your test set to predict your time steps according to the keras prediction model you already built.

2. Substract these predictions from the actual values to get your prediction errors.

3. Plot these in, for example, a histogram to see the usual distribution of errors, like below:

4. According to this plot and statistics, decide on a useful threshold (in this case, e.g. -5 and +7).

5. For your anomaly detection, simply predict the next timestep with your model. Then wait for the actual result of this step and substract it from your prediction. If the difference is bigger then -5 or +7, this is an anomaly.

6. If you find you have too many anomalies, or are missing some, then adjust your thresholds accordingly or retrain your model with different parameters.