I have thousands of very similar datasets that needs to be divided in diagonal way to two groups. for example: enter image description here and enter image description here

I tried to play with the argument of dbscan and optic as epsilon and minPoints and even metric and none of them helped me to divide the data properly to 2 groups.

I only succeed to divide the data using dbscan. If I remove the noise between these groups to make them a complete separate 2 groups, I did it using histogram

j = 1
hist, bin_edges = np.histogram(data, bins=500)
max_bin = np.where(np.amax(hist) == hist)[0][0]
max_noise = bin_edges[max_bin+j]
filtered_indicies = data > max_noise
data = data[filtered_indicies]

these lines remove noise from the data, between the groups and also around it when

j > 1

and that causing me to remove necessary data that I need to reprocess later.

so, I am going back the my main question, how can I know which epsilon, minPoints or other argument of dbscan can help me divide this data properly? or is there maybe a better way then what I presented here above (histogram) to remove the noise between these groups without removing necessary data?


Start with what the parameters mean.

$\varepsilon$ is the search radius around each point. You need this search radius to be small enough that it can't fully "bridge the gap" between the clusters. If the gap width is variable, $\varepsilon$ needs to be small enough to accommodate the narrowest gap. Note that we can exclude the occasional straying point from consideration when determining the gap width as minPoints can handle a small amount of "bleed" into the gap. However, by inspecting the data you can see that making $\varepsilon$ smaller than the smallest gap would cause the points on the far right side to be excluded from the right cluster.

minPoints is the minimum number of points within $\varepsilon$ to point $x$ in cluster $c$ for $x$ to be included in $c$. When we determined the minimum gap width, we allowed a few points from each cluster to cross into the gap. minPoints must be large enough that those stray points don't bridge the clusters. Specifically, if there are $n$ points from cluster $c1$ within $\varepsilon$ of a point in cluster $c2$, minPoints must be greater than $n$. Note that like with $\varepsilon$, setting minPoints large enough to keep the primary clusters separate would cause some points on the edges to be excluded from these clusters.

DBSCAN can't cleanly cluster all points into the two clusters that you want because it connects using local connectivity, not global coherence. Even one bridge from one bunch of points to the other groups them into a single cluster. In order to separate the two clusters, there will be outliers. If outliers are unacceptable, you could join them to the two main clusters using a simple heuristic, such as joining to whichever cluster has a closer center to the outlier. If post-processing is unacceptable, you might want to use a global clustering method such as spectral clustering; you might also have success with mean shift.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.