My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else its Anomaly. Likewise there are thousands of combination. I already tried KModes clustering where number of clusters = unique values in column1. However each cluster do not have equal density hence some bad data with high density is classified as normal and good data with low density is marked anomalous.

I want to have unsupervised algo where I can force it to use column1 as primary criteria for clustering. One with the highest frequency of column2 data for each unique value of column1 is good data. Rest is anomalous. Kindly suggest what would be the best algo and how to approach this problem.

  • $\begingroup$ Clustering is likely to fail. There is little use in relying on heuristics like kmeans when the patterns are that obvious. Juat identify the common values (ABC and XYZ are supposedly frequent) by counting not by clustering, and label everything else as anomalous. $\endgroup$ – Has QUIT--Anony-Mousse Jul 30 '18 at 19:49
  • $\begingroup$ Although I am using KModes (not KMeans), your idea of simple counting and label low frequency pattern as outliers definitely makes sense. I will try it out $\endgroup$ – viral kapadia Jul 31 '18 at 4:17

Your problem is actually a regression problem rather than general clustering, you look for the values far away from the regression line, outliers in the sense of regression. Therefore fit the regression line and filter the values with the largest residual errors which are your "bad" values in the sense of not following the correlation structure given by your two variables.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.