I am dealing with a very strange problem. I have a lot of files. I need to show which files are similar and which one has exception/outliers using its data.

I can show with unsupervised learning using KMeans / DBSCAN or similar ML algorithms for each file. But what would be the approach for such a case? How can I show this record has outliers then group them together?

My datasets are multivariate time series

put on hold as too broad by Stephen Rauch, kbrose, oW_, moh, OmG 16 hours ago

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    What type of files and data do the files contain? Text (txt), numeric (csv) It sounds like you want to identify outliers across all data. How large are the combined files? Do you know how to identify the outlier records or are you looking for groups of data that don't go together? – Skiddles Dec 4 at 15:03
  • They are all in CSV, They have multiple attributes I can identify outliers from a data set, however, I am trying to look for groups which do not go together. – TheTechGuy Dec 4 at 15:07
  • So, is it safe to assume you have some categorical classes that shouldn't be in the same file, or subset of records? – Skiddles Dec 4 at 15:11
  • yes, that could be the case as well, but right now I have data where every file have same attributes / classes – TheTechGuy Dec 4 at 15:13
up vote 1 down vote accepted

If you are looking to find outlier records based and then identify the file containing the records, K-means may be as good a place to start as any, but perhaps you need to join all the files together to train it first. One challenge with K-means and other clustering algorithms is that you have to start with $k$ but sometimes the value of $k$ is hard to determine because the good data can continue to break down into more discrete clusters as $k$ increases. This becomes a problem if your outliers have a high degree of uniqueness because you may need to increase $k$ to catch the outliers, but all you achieve is greater breakdown of the good records.

If you can identify the outliers, you should be able to create a training data set that could be used in a supervised technique like a decision tree. It may take a little longer because you have to label your records, but this may give you a better result in the end.

If you are looking for combinations of records that should not exists in the same file or group of records and assuming you can identify records by some categorical identifier, e.g. transaction type, I would start with a probabilistic approach like apriori, but you may want to check out Naive Bayes based approach. There are some differences in these approaches, but, either one may produce decent results. More generally, I think the analysis you want to perform is $Association\ Rule\ Learning$. This type of technique can be used to derive a statistical probability for the combinations and then you can identify files that contain combinations that have a low probability of co-occurrence.

HTH

  • Thank you so much for the detailed answer. I have used KMeans, but I have analyzed a record and checked outliner. Do you think it make senese to use LSTM autoencoders for such case ? – TheTechGuy Dec 4 at 17:00
  • If your data is time series in nature, an LSTM might make sense but it is more complex and if your data $is$ time series in nature, none of the methods mentioned above are necessarily appropriate. – Skiddles Dec 4 at 17:09
  • my bad, i should have mentioned .All of the data sets are indeed time series. I am updating my question – TheTechGuy Dec 4 at 18:31
  • How can I show this records has outliers then group them together ? – TheTechGuy Dec 5 at 7:44
  • 1
    Your data may be time series in nature, but how you identify outliers is key. If the record contains all the data to identify the outliers, then the methods mentioned above will work. For example, if column c is more than 300% of the value in column c, then you should be fine with non-time series techniques. If you need to know the value of $x_{(i-1,j)}$ to determine the validity of $x_{(i,j)}$, then you need time series techniques. Unfortunately, this problem is a little harder to describe the methodology, but you could start with LSTM if you are comfortable with Neural Networks. – Skiddles Dec 5 at 12:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.