1
$\begingroup$

I have multiple csv files each of which has at least 200MB of data across 12 columns. Each csv file possibly can fall into 4 categories or labels. I am trying to see which clusters each of these files fit. I do not have any code yet. But here is my pseudo-code.

   for file in my_list:
      read data from file
      find the cluster in which this file falls into using k-means or other algorithms
      tag file with the cluster number
   end-for

The result of the clustering would look something like this:

  1. file-1 = cluster-1
  2. file-2 = cluster-2
  3. file-3 = cluster-1
  4. .
  5. .
  6. .
  7. file-n = cluster-4

and so on

Ideally, I would stream the data, for e.g. x,000 or more rows per stream. scikit typically does not handle data that is streamed. Are there other libraries that can take streamed data and achieve what I am looking for?

Many thanks for your suggestions.

$\endgroup$
2
  • $\begingroup$ How do you know there are 4 clusters? Is it possible that one file includes data from different clusters? HINT: Decide/Understand now if your problem is classification or clustering and then follow the correct terminology otherwise you will get confused later. As soon as you answer these questions I will write the answer for you $\endgroup$ Feb 22, 2022 at 11:29
  • $\begingroup$ I don't know if there 4 or n clusters. 4 was just a random pick in my problem statement. The problem I am trying to solve is to find the clusters. I am not looking to classify my data. $\endgroup$ Feb 22, 2022 at 16:36

1 Answer 1

0
$\begingroup$

The approach described in the pseudo-code is flawed for what you want to achieve, because you would be running k-means on every individual file. This means that:

  • The instances in every file are clustered into K groups. Note that this is done based only on the instances in the current file, so even if all the instances in this file are very similar K-means will split them into K clusters.
  • The clusters 1,2,..,K obtained as a result are independent from one file to the other. For example cluster 1 in file 1 may correspond to cluster 3 and 4 in file 2 and no cluster at all in file 3, etc.

You're not going to obtain the kind of output that you expect in this way.

It's important to understand what a clustering algorithm does: it separates the instances of a set by comparing them with each other. Since it's impossible for the clustering algorithm to guess which instances are going to be provided later and how they compare to the current ones, any form of streaming data is incompatible with clustering.

However what is possible is to obtain a clustering model (the centroids with k-means) based on some subset of data, and then apply this model to the rest of the data. This means that the actual clustering (when the centroids are discovered by K-means) takes place only once, then the new instances are just assigned to the closest centroid.

Note: I assume that every file contains multiple instances. In this scenario it's also strange that all the instances in one file belong to the same cluster. In this case I guess that you could use a random sample from every file to obtain the same result.

$\endgroup$
3
  • $\begingroup$ Thank you, Erwan. I agree with everything you say. I was aware that running K-means on each file will cluster that file's data. But I am unable to combine all files as I will run out of RAM. Is there a lazy load K-Means cluster library? $\endgroup$ Feb 23, 2022 at 21:57
  • $\begingroup$ @python_beginner I don't know any. But if you are really sure from the start that all the instances in one file belong to the same cluster, you don't need to use all of them, you can just load a subset from each file. It's a strange case, because in theory every file should be only one instance in this case. $\endgroup$
    – Erwan
    Feb 24, 2022 at 10:47
  • $\begingroup$ Thank you, Erwan! $\endgroup$ Feb 27, 2022 at 5:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.