2
$\begingroup$

I am trying to evaluate some clustering results that a company did for some data but they used an evaluation method for clustering that i have never seen before. So i would like to ask your opinion and obviously if someone is aware of this method it would be great if he/she could explain to me the whole idea.

Clusters have been made to the data set (sample of 250000 rows and 5 features out of 500000 rows) by using k-prototypes as one of the features is categorical. All the combinations of k= 2:10 and lambda = c(0.3,0.5,0.6,1,2,4,6.693558,10) have been made and 3 methods to figure out the best combination have been use.

  1. Elbow method (pick the number of clusters and lambda with the min WSS) enter image description here
  2. Silhouette method pick the number of clusters and lambda with the max silhouette) enter image description here
  3. Decision tree

They build a decision tree for the data and after that they calculated for every different clustering combination the following value: (inverse leaf size weighted within cluster purity)* cluster size/ total obs and the picked the combination which had the max value. (k=10 and lambda=4)

So my question is: Is there such a thing? Can we use the tree to identify which combination will give us higher cluster purity? Also if we can do that can we just use a simple tree without even evaluate how good or bad tree is? And finally, as every single method is giving us different answers how can we decide and pick which one to use to pick the right combinations?

I would really appreciate if someone can help me with that.

Thanks in advance!

| improve this question | |
$\endgroup$
  • $\begingroup$ Decision tree on what exactly? The one categoricial variable? $\endgroup$ – Has QUIT--Anony-Mousse Nov 29 '19 at 20:57
  • $\begingroup$ Note that the "elbow" method is not about picking the smallest WSS - which will always be the maximum k. Instead, it is about finding the point where the curve Stopps dropping fast and instead only drops at a "typical" rate. Here, I'd argue there is no elbow. Which usually indicates that there also is no good clustering result here at all. $\endgroup$ – Has QUIT--Anony-Mousse Nov 29 '19 at 20:59
  • $\begingroup$ Besides computing all these heuristics in a more or less broken way (and apparently not realizing that they even contradict; silhouette for example clearly shows that larger k get much worse at k>=5), did they try with any other means to verify the results?!? $\endgroup$ – Has QUIT--Anony-Mousse Nov 29 '19 at 21:00
  • $\begingroup$ Yeah i know that about the elbow method that is why i included this. So they fit a decision tree with all 5 vars that they used to create the clusters and they added a new target variable that was not the cluster but it was a completed new one which is the original target of the whole project, to be able and cluster our data so we can see if there are clusters with similar behavior to be able and treat them differently regarding the cost that is the new variable. so they pretty much fit a new tree and after that they used the different clusterings to see which one will give them purest nodes $\endgroup$ – Christina Dec 2 '19 at 9:06
  • $\begingroup$ So at the end the pick the combination which was resulting to the purest final nodes. So that is what i can not understand. how they build a decision tree that had a target which was not the cluster so they can use it to check the cluster purity. Also as you mentioned above by using the silhouette method we can see that for k>=5 they clustering is getting worst. So these two methods give us completed different results. $\endgroup$ – Christina Dec 2 '19 at 9:21
1
$\begingroup$

That's a good question.

Is there such a thing? Can we use the tree to identify which combination will give us higher cluster purity?

Clustering and simple decision tree fitting are used together in many cases such as:

  • First, like you mentioned quality of clustering can be measured by using decision tree leafs. I heard this calculation first time (I know some other measures very similar to it) but it makes sence since it still measures how are clusters are distinct from each other and dense.

  • Second (and most used one), fitting a decision tree by assigning cluster labels as class labels. The fit in here should be overfit (training error should be close nearly 0). This let you when you have a new customer (let's say segmentation in e-commerce) you don't have to calculate all distances and find clusters, you just predict the new customer with the tree and assign cluster = segment label to her/him.

Also if we can do that can we just use a simple tree without even evaluate how good or bad tree is?

No, you cannot. The tree fit well (nearly overfit). Since you want to obtain cluster separation rules in your data rather than obtain a model.

And finally, as every single method is giving us different answers how can we decide and pick which one to use to pick the right combinations?

In that stage you should think why are we doing this segmentation? With all possible cluster models try to simulate your business approach and compare results.

Hope it helps!

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks so much for your answer. Can you please let me know as well the following? For the first option where the decision tree is used to measure the quality of the clustering. Which var should be used as the classification label? The cluster labeling? Because in this case the tree is build by using one classification label that it is not used for clustering and it is not the cluster either. For example vars A,b, C and D have been used to create the clusters and the decision tree have been created by E~A+B+C+D instead of cluster ~A+B+C+D $\endgroup$ – Christina Nov 29 '19 at 14:29
  • $\begingroup$ Yes, cluster labels should be used as classification labels. For instance you have k clusters lets say c1, c2,..,ck, than it will be k class multiclass classification. Yes, features are used for both cases. Classification gives you insight about clustering and let you interpret cluster groups. $\endgroup$ – Ilker Kurtulus Nov 29 '19 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.