1
$\begingroup$

I am using LOF ( local Outlier factor) to detect outliers in my data. I get LOF score as outlier distance. this unsupervised learning doesnt help to detect outliers at run time. So I want to use my data points and LOF score to have a supervised model [regression /classification].

My question is which should be approach out of
1) classification ( taking a cutoff of LOF score and after having binary variable , build classification model) and use this classification model to predict outliers at run time.

How does any clustering/anomaly detection( using clustering) is even used at run time ?? Any guidance will help.

$\endgroup$
1
$\begingroup$

You should find an appropriate threshold of the LOF score, above which the point will be considered as an outlier. Typically, a point with score below 1 is considered as normal (non-outlier), but the threshold above 1 at which to consider a point as outlier depends on the dataset.

If you have a dataset with outlier points labeled, then you can use that to fine tune the value of the threshold, such that you get an acceptable tradeoff between false-positive rate (out of points labeled as outliers, fraction of normal points), and recall (out of all actual outliers, fraction which are labeled as outliers).

At run time, if new points are getting added to your dataset, then the local densities and LOF scores across points could change, and these should be recomputed. However, the threshold of LOF score that you determined as optimal should continue to apply, and requires revisiting only over a larger time period.

$\endgroup$
  • $\begingroup$ 1) u mean training continuously. I believe this would not possible. $\endgroup$ – Arpit Sisodia Jun 15 '18 at 15:18
  • $\begingroup$ no we dont have known outlier data. there was no need to LOF/any clustering if i had known outliers, a classification would do!! $\endgroup$ – Arpit Sisodia Jun 15 '18 at 15:19
1
$\begingroup$

You can use LOF online.

For all your training data, compute and store the lrd values.

When you get a new sample, find it's neighbors. Based on the distances to the neighbors and their kids, you can estimate the LOF of the new data point.

If you now do the extra mile to also update the lrd values of the existing points, then you would get an updatable LOF. But that likely isn't worth the effort.

Pokrajac, D., Lazarevic, A., & Latecki, L. J. (2007, March). Incremental local outlier detection for data streams. In Computational Intelligence and Data Mining, 2007. CIDM 2007. IEEE Symposium on (pp. 504-515). IEEE.

$\endgroup$
  • $\begingroup$ @Anony-Mousse- when i get a new data point how can I find it's neighbors. Based on the distances to the neighbors and their kids ? $\endgroup$ – Arpit Sisodia Jun 15 '18 at 15:24
  • $\begingroup$ Use a reverse k nearest neighbor index. $\endgroup$ – Anony-Mousse Jun 16 '18 at 8:05
  • $\begingroup$ ok.. need to read few papers to know it better. Thank @Anony-Mousse $\endgroup$ – Arpit Sisodia Jun 18 '18 at 6:18

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.