Machine learning anomaly detection for continues data

I need to find anomalies in logs created by network monitor tool Bro. Right now I am referring to the example given by Bro Analyze Tool. But the example only uses recent logs.

So I need to make it work with full log meaning to train on past logs and perform anomaly detection in real time on each new log. In the example, as far as I understand, unsupervised machine learning with IsolationForest and KMeans in scikit-learn python are used.

So I have the following question.

For unsupervised machine learning is it possible to store the model with trained data which is already created on previous run?

If possible can I add new data with existing model and then perform anomaly detection in real time?

You can save sklearn models with pickle. This blog post shows how.

• Yes I have found it. So is it possible to append new data with trained model. – Haris Feb 23 '18 at 18:03

Sklearn comes with various ways to save and reuse models, see Model Persistence for more information.

However, it is worth mentioning that not all unsupervised outlier detection models can easily "make the prediction" for the unseen data. predict() method may not be always consistent and visible, such as LOF. As you can see, there is no "predict()" method but "fit()" and "fit_predict()" only. Alternatively, you should use an internal method _decision_function() for LOF in Sklearn instead.

def _decision_function(self, X):
"""Opposite of the Local Outlier Factor of X (as bigger is better,
i.e. large values correspond to inliers).

The argument X is supposed to contain *new data*: if X contains a
point from training, it consider the later in its own neighborhood.
Also, the samples in X are not considered in the neighborhood of any
point.
The decision function on training data is available by considering the
opposite of the negative_outlier_factor_ attribute.


This inconsistency is due to the unsupervised nature of most outlier detection methods. They are no difference for "training" and "test" since the ground truth is missing anyway. For some unsupervised outlier detection methods, saving the "trained" model does not help. It is case by case though. For efficient algorithms, it may be fine to combine the historical data with the new data and refit the model, given they come from the same distribution.

Thus, for different outlier detection methods in Sklearn, you should be careful with model persistence and APIs. If you are looking for more unified APIs to try various anomaly detection methods, I would recommend my python outlier detection toolbox PyOD; the documentation can be found here PyOD documentation