I am trying to evaluate some clustering results that a company did for some data but they used an evaluation method for clustering that i have never seen before. So i would like to ask your opinion and obviously if someone is aware of this method it would be great if he/she could explain to me the whole idea.
Clusters have been made to the data set (sample of 250000 rows and 5 features out of 500000 rows) by using k-prototypes as one of the features is categorical. All the combinations of k= 2:10 and lambda = c(0.3,0.5,0.6,1,2,4,6.693558,10) have been made and 3 methods to figure out the best combination have been use.
- Elbow method (pick the number of clusters and lambda with the min WSS)
- Silhouette method pick the number of clusters and lambda with the max silhouette)
- Decision tree
They build a decision tree for the data and after that they calculated for every different clustering combination the following value: (inverse leaf size weighted within cluster purity)* cluster size/ total obs and the picked the combination which had the max value. (k=10 and lambda=4)
So my question is: Is there such a thing? Can we use the tree to identify which combination will give us higher cluster purity? Also if we can do that can we just use a simple tree without even evaluate how good or bad tree is? And finally, as every single method is giving us different answers how can we decide and pick which one to use to pick the right combinations?
I would really appreciate if someone can help me with that.
Thanks in advance!