2
$\begingroup$

In all the examples that I can see online, people have used a labelled dataset. I however am stuck trying to construct a model to perform anomaly detection on unlabelled dataset (unsupervised anomaly detection).

Problem Statement: To separate the anomalous points from the normal ones. The features available with us are mem_usage, mem_total, cpu_usage_idle, cpu_usage_iowait, etc

Using the anomaly detection, I plan to label the dataset saying whether anomalous or not. A step further would be aggregate each row based on the server the data belongs to and find if the server is healthy, problematic. Even further would be predicting when a server might face a similar problem and thus suggest ways to mitigate the problem.

Any kind of resources (papers, code blogs, videos) are really welcome. I am just starting out and am open to trying out new stuff as well.

| improve this question | |
$\endgroup$
1
$\begingroup$

Do you have sequence data about these metrics? E.g. do they evolve over time and you want to locate anomalies in these sequences?

If so, you can have a look at the tools described here:

https://github.com/rob-med/awesome-TS-anomaly-detection

They are all unsupervised methods, so you don't need anomaly-free data for training.

If you are not dealing with time-series, there are also many unsupervised approaches you can take. This paper offers a good survey on the topic

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152173

| improve this answer | |
$\endgroup$
0
$\begingroup$

https://medium.com/@curiousily/credit-card-fraud-detection-using-autoencoders-in-keras-tensorflow-for-hackers-part-vii-20e0c85301bd

In this part of the series, we will train an Autoencoder Neural Network (implemented in Keras) in unsupervised (or semi-supervised) fashion for Anomaly Detection in credit card transaction data. The trained model will be evaluated on pre-labeled and anonymized dataset.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Nice post. However it uses the fact that labels are known before hand, and trains the auto-encoder only on normal instances. OP data-set has no label, in fact OP wants to label the data set. $\endgroup$ – Esmailian Mar 13 '19 at 9:58
  • $\begingroup$ Yes. There is a bias towards using labelled dataset. Imagine a case where there are no labels. If there is any implementation for such a case, would be happy to know. $\endgroup$ – ripevik Mar 13 '19 at 11:58
0
$\begingroup$

You can refer "Isolation Forest based Anomaly Detection". I think it will help. Reference: Liu et al.,2008. It is an unsupervised anomaly detection.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I have been working on it. Thanks anyway. :) $\endgroup$ – ripevik Mar 14 '19 at 7:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.