2
$\begingroup$

I want to jointly cluster datapoints coming from different datasets (50 datasets with around 2000 points each). I would like to then extract information associated to the datapoints belonging to the different clusters to compare aspects of the datasets.

Now my problem is that if I just place the datapoints originating from the different datasets into the same feature space, too few clusters comprise datapoints coming from all datasets simultaneously, so that after filtering I'm left with too few clusters (I tried with kmeans and similar methods).

My question is: Which is the best way to jointly cluster my points while imposing the condition that a given cluster should contain points from all datasets? The ideal solution would also allow some outliers to not fulfill this condition. The first thing I could think about is to define distances between points and clusters which are updated depending on whether a point belonging to the same dataset is already present in the cluster? Seems too far fetched though.

I would appreciate any ideas, thanks a lot in advance!

Edit: Three of my six features are spatial coordinates and ideally I would also want the clusters to be connected within a given dataset.

Improve this question
$\endgroup$
3
  • $\begingroup$ Have you thought about forming cluster centres using sets comprised of a minimum number of points from each dataset and then force the clusters only to grow ? $\endgroup$
    – image_doctor
    Commented Jul 3, 2015 at 6:36
  • $\begingroup$ Thanks! I think the problem with this would lie in defining the initial clusters centers which I would like to be determined by some variation minimising scheme. $\endgroup$
    – malbert
    Commented Jul 3, 2015 at 12:26
  • $\begingroup$ The implication seems to be that whatever distance metric you use, should not be derived solely from the feature space distance, but have a component dependent on the class of the data, or some data dependent variable weighting of feature distance and class. Without sight of representative data this may prove challenging ? $\endgroup$
    – image_doctor
    Commented Jul 3, 2015 at 19:58

1 Answer 1

Reset to default
1
$\begingroup$

I don't understand the purpose of imposing a condition that requires any cluster to contain at least some (let's say even 1) point from every dataset but at the same time find a solution that also allows that condition to be broken by outliers (I assume these are specific cluster outliers or data set outliers?).

Have you considered implementing an overlapping cluster algorithm? You don't get the full distinctness of a traditional single membership but your might better fit your data to an algorithm that achieves the desired condition. It will probably require some exploration of the data and testing for cluster membership after the fact.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Outliers would be points that in deed don't have correspondence in all datasets. Still I was looking for a way to constrain the clustering for enforcing memberships from all datasets, which in case of the outliers would be too costly. I agree that using a fuzzy clustering might be a good idea, because defining cluster membership could then be assigned using some trade off criterium which on this postprocessing level might be easier to formulate. Thanks! $\endgroup$
    – malbert
    Commented Jul 14, 2015 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.