2
$\begingroup$

I have an interesting problem, and I think my Google is failing me since I can't find the same problem anywhere.

I have a set of individuals. I have 4 different datasets, with (some) to (all) of those individuals in any given set. So if I had 10 people, n=10, then maybe in DS1 I have all 10, DS2 has 7 of them, DS3 8, and DS4 5.

Each dataset DS# has it's own features. So maybe DS1 has 100 features, DS2 has 10, DS3 has 30, and DS4 also has 30.

I want to cluster the individuals who exist in any given dataset. So, if I did 3 clusters total, I'd have 3 clusters for DS1, 3 clusters for DS2, ... , 3 clusters for DS4.

I want a metric that captures how similar the results are. I'm doing this part for exploratory purposes. I'm also considering later incorporating this into a similarity metric for optimization purposes, but that's a further-out idea.

I however cannot find existing work that does something like this. Is there such work someone can point me to?

Much appreciated.

$\endgroup$
0
$\begingroup$

Literature

The closest line of work to your problem is Multi-View Clustering. Each data set DS is considered as a view, and views share a central entity (e.g. an individual).

2004 Multi-View Clustering is a survey on the topic. Also, here are two git projects from two papers (git 1, git 2) on the topic (they are implemented in Matlab and unfortunately are not production-ready).

A shortcut

Since most mainstream clustering methods (e.g. K-means, DBSCAN) accept a custom distance matrix $d(x, y)$ instead of explicit data vectors $x$ and $y$, partiality of features can be handled outside the method.

For example, suppose $x$ and $y$ are both present in views $1$ and $3$ but not view $2$. You may first build their vector by fusing their features from views $1$ and $3$, then computing their distance $d(x, y)$. Do the same for all pairs of individuals based on their common features, and finally feed the distance matrix $d(X, Y)$ to a clustering algorithm. This way, based on your domain knowledge, you can manually assign different weights to features of different views too. For example, you can define the distance as $$d(x, y) := \left \| x[f_1:f_3] - y[f_1:f_3] \right \| + 2\left \| x[f_4:f_5] - y[f_4:f_5] \right \|$$ which implies that the difference in view $3$ with features $f_4$ and $f_5$ is twice as important as the difference in view $1$ with features $f_1$ to $f_3$.

Comparing two partitions

If views are clustered separately, two partitions $p_1$ and $p_2$ respectively over views $1$ and $2$ can be compared using metric Normalized Variation of Information (NVI). If partitions are the same, it outputs $0$, and if they are statistically independent, it outputs $1$. A similar measure (that is not a metric) is Normalized Mutual Information (NMI).

If all individuals are not present in both partitions, either (1) remove non-shared individuals from both partitions, or (2) individuals in $p_1$ that are not present in $p_2$ can be assigned to an imaginary cluster $misc_2$ in $p_2$, the same for those in $p_2$ that are not in $p_1$ using $misc_1$ in $p_1$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.