Clustering multiple csv files that cannot fit in RAM

Question

I have multiple csv files each of which has at least 200MB of data across 12 columns. Each csv file possibly can fall into 4 categories or labels. I am trying to see which clusters each of these files fit. I do not have any code yet. But here is my pseudo-code.

   for file in my_list:
      read data from file
      find the cluster in which this file falls into using k-means or other algorithms
      tag file with the cluster number
   end-for

The result of the clustering would look something like this:

1. file-1 = cluster-1
2. file-2 = cluster-2
3. file-3 = cluster-1
4. .
5. .
6. .
7. file-n = cluster-4

and so on

Ideally, I would stream the data, for e.g. x,000 or more rows per stream. scikit typically does not handle data that is streamed. Are there other libraries that can take streamed data and achieve what I am looking for?

Many thanks for your suggestions.

Answer 1

The approach described in the pseudo-code is flawed for what you want to achieve, because you would be running k-means on every individual file. This means that:

• The instances in every file are clustered into K groups. Note that this is done based only on the instances in the current file, so even if all the instances in this file are very similar K-means will split them into K clusters.
• The clusters 1,2,..,K obtained as a result are independent from one file to the other. For example cluster 1 in file 1 may correspond to cluster 3 and 4 in file 2 and no cluster at all in file 3, etc.

You're not going to obtain the kind of output that you expect in this way.

It's important to understand what a clustering algorithm does: it separates the instances of a set by comparing them with each other. Since it's impossible for the clustering algorithm to guess which instances are going to be provided later and how they compare to the current ones, any form of streaming data is incompatible with clustering.

However what is possible is to obtain a clustering model (the centroids with k-means) based on some subset of data, and then apply this model to the rest of the data. This means that the actual clustering (when the centroids are discovered by K-means) takes place only once, then the new instances are just assigned to the closest centroid.

Note: I assume that every file contains multiple instances. In this scenario it's also strange that all the instances in one file belong to the same cluster. In this case I guess that you could use a random sample from every file to obtain the same result.

Answer 2

One solution is to reduce the size of the input data by using Principle Component Analysis (PCA), or a similar technique, to reduce its dimensionality.

You would loop through your input files one-by-one, run a PCA (for example) and save the resulting output. Then you can conduct k-means on your new reduced dataset.

This is a common technique in NLP given the large and sparse nature of input data in that field.