4
$\begingroup$

I have the following challenge: We have the web logs of a platform where people can download publications and we need to detect anomalies.

From time to time and only by chance we observe spikes in usage over a day or so, where there are many more downloads than usually. Often these spikes are caused by the same IP address or several from the same IP address range. Also, these requests are for the same publication (as identified by the same URL).

I now wonder how we can identify anomalies like this automatically. Taking into account that sometimes, an item is not downloaded for several days or weeks, so there is not necessarily a lot of "normal" download data available.

The business wants us to detect these anomalies, qualify them (are they fraudulent or legit) and exclude them from the overall results if fraudulent.

What would be the best approach to tackle this problem?

$\endgroup$

1 Answer 1

4
$\begingroup$

Since you've used the word 'Automatically', I assume that you're looking for an unsupervised method.

Unsupervised Anomaly Detection techniques do not need training data. They are based on two basic assumptions. First, they presume that most of the network connections are normal traffic and only a very small traffic percentage is abnormal. Second, they anticipate that malicious traffic is statistically various from normal traffic. According to these two assumptions, data groups of similar instances which appear frequently are assumed to be normal traffic, while infrequently instances which considerably various from the majority of the instances are regarded to be malicious.

The most common unsupervised algorithms are:

  • K-Means
  • Self-organizing maps (SOM)
  • C-means
  • Expectation-Maximization Meta algorithm (EM)
  • Adaptive resonance theory (ART)
  • Unsupervised Niche Clustering (UNC)
  • One-Class Support Vector Machine

I suggest you to take a look into all of the above methods and choose the one that fits your problem. There are also more advanced and modern approaches which I don't suggest (at the beginning). If the above methods didn't solve your problem, then use the more advanced methods.

$\endgroup$
3
  • $\begingroup$ Thank you @amirhossein. Is the order of the algorithms you listed the perceived appropriateness or is it non-significant? Is there one you would recommend starting with, especially for a beginner? $\endgroup$
    – jfix
    Dec 5, 2022 at 12:58
  • 1
    $\begingroup$ No they are not in order. I would start at K-Means but each algorithm has its own ups and downs, a quick look on all of them would be better before starting to implementing them on your problem. $\endgroup$ Dec 5, 2022 at 16:09
  • $\begingroup$ Here is an interesting exploration of the K-Means algorithm: k-means-explorable.vercel.app $\endgroup$
    – jfix
    Dec 20, 2022 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.