2
$\begingroup$

In all the examples that I can see online, people have used a labelled dataset. I however am stuck trying to construct a model to perform anomaly detection on unlabelled dataset (unsupervised anomaly detection).

Problem Statement: To separate the anomalous points from the normal ones. The features available with us are mem_usage, mem_total, cpu_usage_idle, cpu_usage_iowait, etc

Using the anomaly detection, I plan to label the dataset saying whether anomalous or not. A step further would be aggregate each row based on the server the data belongs to and find if the server is healthy, problematic. Even further would be predicting when a server might face a similar problem and thus suggest ways to mitigate the problem.

Any kind of resources (papers, code blogs, videos) are really welcome. I am just starting out and am open to trying out new stuff as well.

$\endgroup$

4 Answers 4

1
$\begingroup$

Do you have sequence data about these metrics? E.g. do they evolve over time and you want to locate anomalies in these sequences?

If so, you can have a look at the tools described here:

https://github.com/rob-med/awesome-TS-anomaly-detection

They are all unsupervised methods, so you don't need anomaly-free data for training.

If you are not dealing with time-series, there are also many unsupervised approaches you can take. This paper offers a good survey on the topic

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152173

$\endgroup$
0
$\begingroup$

https://medium.com/@curiousily/credit-card-fraud-detection-using-autoencoders-in-keras-tensorflow-for-hackers-part-vii-20e0c85301bd

In this part of the series, we will train an Autoencoder Neural Network (implemented in Keras) in unsupervised (or semi-supervised) fashion for Anomaly Detection in credit card transaction data. The trained model will be evaluated on pre-labeled and anonymized dataset.

$\endgroup$
2
  • $\begingroup$ Nice post. However it uses the fact that labels are known before hand, and trains the auto-encoder only on normal instances. OP data-set has no label, in fact OP wants to label the data set. $\endgroup$
    – Esmailian
    Mar 13, 2019 at 9:58
  • $\begingroup$ Yes. There is a bias towards using labelled dataset. Imagine a case where there are no labels. If there is any implementation for such a case, would be happy to know. $\endgroup$
    – ripevik
    Mar 13, 2019 at 11:58
0
$\begingroup$

You can refer "Isolation Forest based Anomaly Detection". I think it will help. Reference: Liu et al.,2008. It is an unsupervised anomaly detection.

$\endgroup$
1
  • $\begingroup$ I have been working on it. Thanks anyway. :) $\endgroup$
    – ripevik
    Mar 14, 2019 at 7:45
0
$\begingroup$

I used the Isolation Forest Algorithm to identify the anomaly points. You can take a look.. https://github.com/muratyldrim/CatchME

$\endgroup$
2
  • $\begingroup$ Provide context for links Links to external resources are encouraged, but please add context around the link so your fellow users will have some idea what it is and why it’s there. Always quote the most relevant part of an important link, in case the external resource is unreachable or goes permanently offline. Links to other websites should always be helpful, but avoid making it necessary to click on them as much as possible. $\endgroup$
    – user51855
    Dec 4, 2022 at 17:05
  • $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$
    – Ethan
    Dec 4, 2022 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.