I've been assigned to the following task:
I was given 1,000,000 data points and was asked to create a sort of distance matrix and to cluster the rows. So this matrix is 1,000,000 x 1,000,000 which is obviously way to large to fit on my poor 8GB of RAM.
I'd like to ask for some tips of how to handle this kind of data.
I'd like to choose maybe 100,000 data points at random and cluster their distances instead hoping they represent the entirety of the data.. even so this seems like a hard task.
So what kinds of clustering methods could work here? If I can't feed all my data at once to some algorithms which usually can handle lots of data such as hierarchical clustering or DBscan what options do I have left?