I am a newbie to data science with a typical problem. I have a data set with metric1, metric2 and metric3. All these metrics are interdependent on each other. I want to detect anomalies in metric3. Currently, I am using Nupic from numenta.org for my analysis and it doesn't seem to be effective. Is there any ML library which can detect anomalies in multiple parameters?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ I think you should clarify your problem a bit. So you think metrics1 and metrics2 predict metrics3, and want to know when metrics3 doesn't match the prediction well? that's just a regression problem. $\endgroup$– Sean OwenNov 3, 2014 at 8:28
-
$\begingroup$ @SeanOwen : Yes absolutely. metrics1 and metrics2 predict metric3. for example, temperature and pressure metrics predict a metric called constant for a fixed mass of gas. A change in the metric constant means there an anomaly detected in the mass of gas and an action has to be taken. Similarly, at times, we want to predict temperature or pressure metrics from metric constant as well. In the end, all these 3 metric values are streamed to the algorithm. $\endgroup$– codejammerNov 3, 2014 at 13:23
-
$\begingroup$ Take a look at information I've shared in this related answer. I hope that it'll be helpful. $\endgroup$– Aleksandr BlekhMar 6, 2015 at 4:29
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
0
One way to use both metric1 and metric2 in order to find anomalies in metric3 is to consider residual analysis.
In your case, this would require, creating a predictive model with metric1 and metric2 as the predictors and metric3 as the response variable.
Then, calculate the residuals for metric3 as its predicted value subtracted from its true value. Now, you can report the all members of the lowest decile [or any other percentile] as one kind of an anomaly and all the members of the highest decile [or any other percentile] as another kind of an anomaly.
$\begingroup$
$\endgroup$
If you label metric 3 as $x_3 = \{1,0\}$, where $1$ means it is an anomaly, this becomes a logistic regression problem where $\mathbb{P}(X_3 = 1) = logit(\beta_0 + \beta_1 x_1 + \beta_2 x_2)$.
