Is anyone aware of any open source / python packages for Netflow Anomaly detection ? I found some on github but anyone who has more experience with it. please advise.
I could not find any specific package in the internet that does the same thing as netflow anomaly detection, but I found out how to train some standard algorithms that solve the same problem. My understanding is that the problem is "network intrusion detection" or "anomaly detection in network traffic to detect network intrusion"
So creation of a model can be divided into several parts:
- choosing an appropriate algorithm
- understand how to transform the data for the algorithm
- finding the data set
- training the model
The above tasks in more detail:
- algorithm: k-NN or Random Forest. According to the paper Machine Learning DDoS Detection for Consumer Internet of Things Devices k-nearest neighbor is a pretty precise algorithm in network anomaly detection. Nearest neighbor algorithms are present in scikit-learn python package (link). Random forest classifier performed even better. scikit-learn also has a random forest classifier (link).
How k-NN for DDoS works is explained in this paper: Proactive Detection of DDoS Attacks Utilizing k-NN Classifier in an Anti-DDos Framework.
According to the above mentioned papers and other sources the following datasets are used for training:
- KDCup1999 This is a network intrusion database
- MIT Lincoln Lab, 2000, DARPA intrusion detection scenario specific datasets Contains two DoS attack scenarios
- Reasearchgate question on available datasets
On this link you will find how to train a k-nearest neighbor in scikit-learn.
Here is an example code from the link:
>>> X = [, , , ] >>> y = [0, 0, 1, 1] >>> from sklearn.neighbors import KNeighborsClassifier >>> neigh = KNeighborsClassifier(n_neighbors=3) >>> neigh.fit(X, y) KNeighborsClassifier(...) >>> print(neigh.predict([[1.1]]))  >>> print(neigh.predict_proba([[0.9]])) [[0.66666667 0.33333333]]