Suppose I have a dataset containing two very similar classes of data. By similar, I mean that the 'distance' between these two classes is very small. For example, one instance in Class 1 is the sum of one instance in Class 2 and some noise. If the SNR is high, we can say these two instances are similar. Because of the similarity, the dataset is inseparable. I am wondering if there exists any effective clustering algorithm that can work. Thank you very much！
As you point out, the problem is not on the clustering algorithm, but on the features. So the question comes to the particular data you might be dealing with.
As an example, say you want to cluster different kind of animals. It is in general much easier to tell an elephant from a horse apart. But if you want to distinguish between races of horses, it gets much harder. But the bottleneck lies (mostly) on the features.