1
$\begingroup$

I have a particular dataset consisting of 50k elements with 40 features each. I want to try to cluster the data as it is, without any dimensionality reduction. The main algorithm I am considering is the DBSCAN since is the more versatile and I can accept some poits to result as noise. However how can I judge if the clustering is "significant" since I can't plot the clusters in comparison to the data? Tring to select the paremeters for the DBSCAN I've done a k-nn analysis, but the results worried me. For example, the following is the third nearest neighbor plot. 3-NN As you can see the distances (y-axis) are pretty much "uniform" along the x-axis (object). Does it mean that the data is somehow uniformly sparse and, in this conditions, clustering is useless?

P.S. I have tried to cluster the data anyway, in particulary using epsilon around 2-2.5 and for different values of min_sample. The silhouette score however results very low, about 0.11, and the noiseless fraction of points is about 80%.

Improve this question
$\endgroup$
8
  • $\begingroup$ Your results do not make sense. Judging by the plot, at epsilon=2, almost everything should be connected. It's a way too large value for this k. $\endgroup$
    – Has QUIT--Anony-Mousse
    May 30, 2018 at 6:46
  • $\begingroup$ Sorry but I don't get the point. Taking 2 as a distance only means that i can find 3 neighbors for almost all points. In my opinion this doesn't mean that everything should be connected. Can you explain me more? $\endgroup$
    – M. Fabio
    May 30, 2018 at 7:07
  • $\begingroup$ If almost every point has at least x neighbors (and many will have much more than x) then almost everything will have a path to almost everything else. As the name suggests, epsilon should usually be a rather small value. Probably, your minpts is too small. $\endgroup$
    – Has QUIT--Anony-Mousse
    May 30, 2018 at 7:11
  • $\begingroup$ Following your advise I scanned the data with DBSCAN with a range of parameters for epsilon: [0.1, 2] and for minpts: [2, 60]. The results are very poor, none of the cases reached a silhouette score of 0.1. $\endgroup$
    – M. Fabio
    May 30, 2018 at 9:10
  • $\begingroup$ Silhouette is never high on data with noise, by definition. Don't rely on it. It also isn't good for arbitrary shaped clusters, or clusters of varying sizes, because it consider the distance to each cluster point. $\endgroup$
    – Has QUIT--Anony-Mousse
    May 30, 2018 at 22:43

1 Answer 1

Reset to default
0
$\begingroup$

You can look at several clustering metrics (e.g., Silhouette Coefficient, Calinski-Harabasz index, or Davies-Bouldin index) and see if they revealing patterns in your clusters.

You can look at relative performance by trying different hyperparameters for DBSCAN and comparing DBSCAN to other clustering techniques.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.