0
$\begingroup$

My goal is to develop an anomaly detection model with Isolation Forest in order to distinguish between normal and anomalous IPs by analyzing the web access logs and extracting some features in batches of 2 minutes. These features include how many times a specific IP did a request in the batch of 2 minutes, etc.

An example of training data is the following:

      0  1     2  3
0     4  2  37.5  1
1     1  1   0.0  0
2     2  1   0.0  0
3    39  1   0.0  0
4    34  1   0.0  0
5   158  1   0.0  0
6    15  1   0.0  0
7    61  1   0.0  0
8    13  1   0.0  0
9     1  1   0.0  0
10    1  1   0.0  0
11    2  2  75.0  1
12    8  1   0.0  0
13    1  1   0.0  0
14    1  1   0.0  0
15    1  1   0.0  0
16    1  1   0.0  0
17  274  1   0.0  0
18   40  1   0.0  0

An example of test data that I am using to predict is the following:

     0  1    2  3
0  392  1  0.0  0

Although I would expect this kind of input to be characterized as anomalous, since the first feature is larger than every correspondent feature in the training data, it is characterized as normal. My question is why something like that happens? I use contamination=0.1.

$\endgroup$
6
  • $\begingroup$ Do you happen to have a higher value in the training dataset for the 1st feature value (392). As the training dataset you posted is just a sample dataset, we don't know for sure. $\endgroup$
    – Polymath
    Sep 7, 2022 at 7:31
  • $\begingroup$ @Polymath No, this is the exact dataset that I used for this simple test. There does not exist higher number. $\endgroup$
    – Kosmylo
    Sep 7, 2022 at 7:33
  • $\begingroup$ want to check one thing. Could you combine the test set with the training and then run the IF model, and see, if it identifies it as an Anamoly, contamination could be varied from Auto to .1 variations $\endgroup$
    – Polymath
    Sep 7, 2022 at 8:40
  • $\begingroup$ @Polymath If I add the test data into the train data and then run the model, it identifies it as anomaly. This is weird. Why does this happen? $\endgroup$
    – Kosmylo
    Sep 7, 2022 at 14:49
  • $\begingroup$ IF is an unsupervised learning algorithm, so there is "no training/ learning" taking place. It just creates the random decision trees with taking random features with thresholds to isolate data points. So, your test data point, having a higher value (392, for the first feature) would get isolated due to the threshold value $\endgroup$
    – Polymath
    Sep 7, 2022 at 14:59

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.