0
$\begingroup$

I have a dataset containing 150k rows and 10 columns. After clustering, I would like to get clustering metrics. Below are lists of metrics that I would like to use>

 "C_index", "Calinski_Harabasz", "Davies_Bouldin", "S_Dbw", "Silhouette"

I have a problem. I've tried getting values using clusterCrit (from R) and sklearn (from python), but when I tried to get each value by using clusterCrit, it took a huge amount of time, so I couldn't get it.

I think this is because of the large data (my RAM is 8GB). Is there any way for me to get these values?

$\endgroup$
1
$\begingroup$

Just use a subsample of your data.

There is little use in evaluating (nor clustering, usually) the entire data set. It's highly redundant. Trying to scale this to the entire data set is a waste of time.

$\endgroup$
  • $\begingroup$ Thanks for the reply, as I'm new to this, I want to ask about your answer specifically. I have 2 questions. 1. When I do clustering with lot of methods (Spectral, Agglomerative, DBSCAN, OPTICS, Kmeans, ...), if I fit sample data and predict other rest data, can I consider that whole result as validated result? 2. Also, when I divide the clustering result into sample result and get clustering metrics, can I say it is meaningful to all data?? $\endgroup$ – Hyeon May 20 '19 at 4:29
  • $\begingroup$ How can you say it is meaningful even if you had used all data? All data is also just a sample. The purpose of clustering is not to maximize some index. $\endgroup$ – Has QUIT--Anony-Mousse May 20 '19 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.