Plots of data using DBSCAN algorithm not making sense

Question

I am using clustering for my data. Since the DBSCAN algorithm will also tell me an estimate of clusters that I can use, I have used DBSCAN. I have tried for the eps=[0.123,1,2] and min_smaples=[2,10,...60]. The print satatement in the below code prints 714 which is equal to the number of data-samples(rows). The code looks like:

dbscan = DBSCAN(eps=1, min_samples = 4)
clusters = dbscan.fit_predict(df)
print(len(clusters))
plt.scatter(df.iloc[:, 0], df.iloc[:, 1], c=clusters, cmap="plasma")
plt.xlabel("Feature 0")
plt.ylabel("Feature 1")

Sample plots for some of the different min_samples is shown below:

The parameters for the above plots are given below in the same pattern.

1. EPS=1, MIN_SAMPLES=2, EPS=1, MIN_SAMPLES=10
2. EPS=1, MIN_SAMPLES=20, EPS=1, MIN_SAMPLES=40

None of these plots are making sense to me from a clustering perspective. From this, I am forced to conclude that I cannot use clustering for the given data or I am doing it wrong. So, I need help with insights into the weird appearance of the above plots.

Any help is appreciated

Answer 1

Dont try to visually confirm it.

You are plotting your clustering resutls in ONLY two dimensions and you expect that all of the information is in these two dimesnions. That is very unlikely. If you plot 3 dimensions you will see even more seperability and it will make a bit more sense. In any case you need a metric for example Silhouette that tells you how well you clustered. Visualisation is just a sanity check if you know your features already.

Answer 2

Don't try to find parameters by brute force.

Instead, analyze your data. The choice of minpts is application driven - how noisy your data is, how many points you require for a point to be considered important. Based on this, you can choose epsilon based on the k-distance plot.

Try projecting your data into different views when you have multiple dimensions.

Also try different preprocessing. You seem to have scaled your data to 0:1, but is this the right scaling to capture similarity? If your distance does not capture similarity, then DBSCAN will not work because it relies on your distance function...