0
$\begingroup$

I have a dataset which consists of 2 types of items. One type (Type A) is easily matchable, and there are several groups within this set I would like to cluster. The second type (Type B) is randomly scattered about, and I would like to cluster all of these into a single bucket. I know how many clusters of Type A to expect.

The data items are all vectors of length 10ish (edit - it turns out I may have 100 elements in each vector! , so there are plenty of dimensions to play with. Even so, I expect there will be some Type B items which "fall into" one of the Type A buckets, and I can accept that.

A toy example:

import numpy as np
import random

observation_length = 10

# create some sample templates - 
data_templates = [
    np.random.random(size=(observation_length,)), # Type A1
    np.random.random(size=(observation_length,)), # Type A2
    None                                          # Type B - actual values created below
]

# generate some sample data
observations = []
for _ in range (100):
    observation = random.choice(data_templates)
    if observation is None:
        observation = np.random.random(size=(observation_length,))
    # in reality the observations will also have measurement noise
    
    observations.append(observation)

How would I cluster the observations into 3 buckets in such a system?

$\endgroup$

2 Answers 2

1
$\begingroup$

Looks like a case suitable for (H)DBSCAN. It assumes there exists a foreground (real/interesting data to be clustered) and background (noise). In your case group As would be foreground and B the background.

$\endgroup$
0
$\begingroup$

It might be useful to frame that problem as hierarchical clustering. In particular, use a "top-down" divisive approach. All observations start in one cluster, then clusters are split recursively down the hierarchy. The first level is "A" / "not A", then the second level is the split of "A" into "A1" and "A2". The divisive split decision will be based on a measure of dissimilarity between the observations.

$\endgroup$
1
  • $\begingroup$ Thanks - would that still work if I have say 20 "A groups" and one "not A" group? It feels unbalanced, but perhaps that is not a problem? $\endgroup$ Jul 18, 2023 at 10:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.