I am trying to setup a big data infrastructure using Hadoop, Hive, Elastic Search (amongst others), and I would like to run some algorithms over certain datasets. I would like the algorithms themselves to be scalable, so this excludes using tools such as Weka, R, or even RHadoop. The Apache Mahout Library seems to be a good option, and it features algorithms for regression and clustering tasks.
What I am struggling to find is a solution for anomaly or outlier detection.
Since Mahout features Hidden Markov Models and a variety of clustering techniques (including K-Means) I was wondering if it would be possible to build a model to detect outliers in time-series, using any of this. I would be grateful if somebody experienced on this could advice me
- if it is possible, and in case it is
- how-to do it, plus
- an estimation of the effort involved and
- accuracy/problems of this approach.