0
$\begingroup$

I am working on a mall customer segmentation dataset (5 features, 200 rows) using clustering. This dataset does not have any ground truth labels. I had a few doubts regarding clustering:

  1. Can I use model selection in clustering using the silhouette score? - Since my dataset does not have any ground truth labels, I read on the sklearn documentation that you can use Silhouette score to evaluate the performance of the model. Can I use different clustering techniques (like K Means, DBSCAN, Mean shift, etc.) and select the model with the highest silhouette score? The idea is sort of similar to how we do model selection in supervised learning except in the latter we use cross validation.

  2. How do I detect overfitting in clustering? Since the dataset has no labels, I cannot think of a way to identify if the model is overfitting the data.

  3. How do I plot the final clusters when my dataset has more than 2 dimensions? I have seen a lot of visualizations around clustering (like the one below):

enter image description here

Should I use PCA to reduce the features to 2 and then plot the clusters? or is there another way to do this?

Improve this question
$\endgroup$
4
  • $\begingroup$ Just remember applying some scaling technique such as Standard or MinMax prior clustering, even you can use some manifold technique such as tSNE as preprocessing step since it has been shown to work for finding hidden structures on our data (stats.stackexchange.com/questions/263539/…) Additionally, I recommend using soft clustering techniques such as Gaussian Mixtures since in this way you will get a probability that a given point belongs to a cluster rather than only the label. It might be useful when making decisions on your data $\endgroup$
    – Multivac
    Commented Aug 28, 2020 at 15:05
  • $\begingroup$ When using Gaussian Mixtures, isn't one of the assumptions that the data is actually a mixture of different gaussian distributions? So shouldn't we transform the data using PowerTransformer() (scikit-learn.org/stable/modules/generated/…) to make the data more Gaussian like? $\endgroup$
    – Aastha Jha
    Commented Aug 28, 2020 at 16:03
  • $\begingroup$ Correct, but I recommend using QuantileTransformer instead, in that way you guarantee your data is Normal Multivariate distributed scikit-learn.org/stable/modules/generated/… $\endgroup$
    – Multivac
    Commented Aug 28, 2020 at 18:59
  • $\begingroup$ Note that Silhouette Score is not a good measure for DBSCAN $\endgroup$
    – Dr Nisha Arora
    Commented Oct 20, 2021 at 15:01

1 Answer 1

Reset to default
0
$\begingroup$

To answer your initial question, yes you can use silhouette score with different clustering methods. You could also use the Davies-Bouldin Index or the Dunn Index.

Regarding over-fitting, (this is my personal suggestion) but you could train the model n times on different types of the same data to see if there clustering is the same even though the values are changed. Short example: If you have to cluster 5 apples and 6 oranges, the cluster should be the same for 10 apples and 12 oranges. You can find a bit more detail on this here: https://datascience.stackexchange.com/a/20292/103857

For your third query: Calculate distances between data points, as appropriate to your problem. Then plot your data points in two dimensions instead of fifteen, preserving distances as far as possible. This is probably the key aspect of your question. Read up on multidimensional scaling (MDS) for this. Finally, color your points according to cluster membership.

(source for third query: https://stats.stackexchange.com/a/173823)

Regarding pca, its subjective. PCA works well with high correlation. If your dimensions are like apples and oranges then your directly effecting your models performance, so do keep that in check. A bit of eda would help before you dive into that.

Improve this answer
$\endgroup$
3
  • $\begingroup$ For point one, remember that most of the clustering algorithms require the number of clusters to be defined prior, so if you are selecting across different algorithms it might be misleading if you do not use the "optimal" number of clusters, besides it would be difficult to discriminate between a model that finds the number of clusters by itself like DBSCAN vs Kmeans $\endgroup$
    – Multivac
    Commented Aug 28, 2020 at 15:09
  • $\begingroup$ True, plus actually using a metric like silhouette score, Calinski-Harabasz Index or DB index can be a bit misleading since it is higher for convex clusters but since the dataset does not have any labels, the sklearn documentation only suggests these metrics $\endgroup$
    – Aastha Jha
    Commented Aug 28, 2020 at 16:07
  • $\begingroup$ Oh okay!! Thanks for pointing that out! I'll keep that in mind next time something like this comes up! $\endgroup$
    – Academic
    Commented Aug 29, 2020 at 0:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.