10
$\begingroup$

I am trying to setup a big data infrastructure using Hadoop, Hive, Elastic Search (amongst others), and I would like to run some algorithms over certain datasets. I would like the algorithms themselves to be scalable, so this excludes using tools such as Weka, R, or even RHadoop. The Apache Mahout Library seems to be a good option, and it features algorithms for regression and clustering tasks.

What I am struggling to find is a solution for anomaly or outlier detection.

Since Mahout features Hidden Markov Models and a variety of clustering techniques (including K-Means) I was wondering if it would be possible to build a model to detect outliers in time-series, using any of this. I would be grateful if somebody experienced on this could advice me

  1. if it is possible, and in case it is
  2. how-to do it, plus
  3. an estimation of the effort involved and
  4. accuracy/problems of this approach.
$\endgroup$
3
  • 1
    $\begingroup$ This is too vague to be answered. Time series are too different to just throw k-means on them and get out anything useful. It heavily depends on your data. $\endgroup$ Oct 17, 2014 at 12:14
  • 1
    $\begingroup$ For outlier detection, have a look at the algorithms in ELKI. That seems to be the most complete collection of outlier detection. $\endgroup$ Dec 9, 2014 at 22:37
  • $\begingroup$ The newer Elasticsearch versions have time series anomaly detection built in (I think you have to buy the X-Pack). I am not sure what algorithms they are using but it might be worth investigating an off-the-shelf solution. $\endgroup$
    – tom
    Nov 15, 2017 at 22:04

2 Answers 2

7
$\begingroup$

I would take a look at t-digest algorithm. It's been merged into mahout and also a part of some other libraries for big data streaming. You can get more about this algorithm particularly and big data anomaly detection in general in next resources:

  1. Practical machine learning anomaly detection book.
  2. Webinar: Anomaly Detection When You Don't Know What You Need to Find
  3. Anomaly Detection in Elasticsearch.
  4. Beating Billion Dollar Fraud Using Anomaly Detection: A Signal Processing Approach using Argyle Data on the Hortonworks Data Platform with Accumulo
$\endgroup$
2
  • $\begingroup$ How does t-digest compare to the p-square algorithm? $\endgroup$
    – David Marx
    Oct 17, 2014 at 17:16
  • $\begingroup$ Thanks for the answer: this is a simple model to compute extreme quantiles, and I think it will fit my needs. However for more complex time-series that do not have a nearly stationary distribution this approach may fail, and that's when I think we would need something adaptive such as a Markov chain. $\endgroup$
    – doublebyte
    Oct 20, 2014 at 9:32
0
$\begingroup$

You can refer to my response related to h2o R or Python anomaly detection method in stackexchange,since that is scalable too.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.