I have 1 million rows with 20 attributes to do hierarchical clustering. When I want to build a distance matrix on this data by
dist() in R, it says that it needs 5 TB memory. I have these approaches:
- Reduce the number of rows by sampling
- Change the method of clustering
Now, do you suggest another approach? And I have an idea, I think if I reduce the accuracy of the values and then doing "group by", then I can remove duplicated rows and have a new column with the count of duplicates for each row. Is there any R package that can do hierarchical clustering with these data?
"group by": count number of duplicated rows and add a column that say how many times this row was duplicated in source.