6
$\begingroup$

My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else it's Anomaly. Likewise, there are thousands of combinations. I already tried KModes clustering where a number of clusters = unique values in column1. However, each cluster does not have equal density hence some bad data with high density is classified as normal and good data with low density is marked anomalous.

I want to have unsupervised algo where I can force it to use column1 as the primary criteria for clustering. One with the highest frequency of column2 data for each unique value of column1 is good data. Rest is anomalous. Kindly suggest what would be the best algo and how to approach this problem.

Improve this question
$\endgroup$
2
  • $\begingroup$ Clustering is likely to fail. There is little use in relying on heuristics like kmeans when the patterns are that obvious. Juat identify the common values (ABC and XYZ are supposedly frequent) by counting not by clustering, and label everything else as anomalous. $\endgroup$
    – Has QUIT--Anony-Mousse
    Jul 30, 2018 at 19:49
  • $\begingroup$ Although I am using KModes (not KMeans), your idea of simple counting and label low frequency pattern as outliers definitely makes sense. I will try it out $\endgroup$
    – viral kapadia
    Jul 31, 2018 at 4:17

3 Answers 3

Reset to default
0
$\begingroup$

One option is counting patterns. Then define less common occurring patterns as an anomalies.

The counting approach is deterministic, whereas clustering is probabilistic. It might solve your problem. If not, it will at least provide summary statistics and a baseline model.

Improve this answer
$\endgroup$
0
$\begingroup$

You can create a map (dataset) of the normal combinations and then classify new samples as anomalous if they are not found in the dataset of the normal combinations. You do not need to use any machine learning for your problem. You can, but you do not need to. If you insist on using machine learning you can build a regression model that predicts the second value (XYZ) based on the first value (ABC) and if the real second value does not equal the predicted value, you can mark it as anomalous. Again, this is unnecessary, so I would recommend to just use a map and a simple script.

Improve this answer
$\endgroup$
-1
$\begingroup$

Your problem is actually a regression problem rather than general clustering, you look for the values far away from the regression line, outliers in the sense of regression. Therefore fit the regression line and filter the values with the largest residual errors which are your "bad" values in the sense of not following the correlation structure given by your two variables.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.