5
$\begingroup$

I have two sets of topics obtained from two different sets of news paper articles.

In other words, Cluster_1 = ${x_1, x_2, ..., x_n}$ includes the main topics of 'X' news paper set and Cluster_2 = ${y_1, y_2, ..., y_n}$ includes the main topics of 'Y' news paper set.

Now I want to find clusters in the two sets that are similar/related by considering the cluster attributes as given in the example below.

Example 1,
**X1 in Cluster_1** is mostly similar/related to **Y2 in Cluster_2**
**X2 in Cluster_1** is mostly similar/related to **Yn in cluster_2**
and so on.

Example 2:
News about Yet in Cluster_1 is mostly similar/related to News about Science in Cluster_2
News about Floods in Cluster_1 is mostly similar/related to News about Rains in Cluster_2

Since, I am dealing with two separate sets of clusters, what would be a suitable measurement/method I can use to connect the clusters in the two different sets?

$\endgroup$
  • $\begingroup$ Are these two different groupings of the same observations? In the same space? Same dimensional space? What does "most similar/related" mean to you in this context? $\endgroup$ – Thomas Cleberg Aug 17 '17 at 2:31
  • $\begingroup$ Thank you for the comment. I edited the question by including the missing information you have mentioned. $\endgroup$ – Volka Aug 17 '17 at 2:48
  • $\begingroup$ Are your clusters the result of topic modeling with something like Latent dirichlet allocation over two newspapers, and you're wondering if you can compare the topics between the two newspapers? $\endgroup$ – Thomas Cleberg Aug 17 '17 at 3:26
  • $\begingroup$ Yes, you are correct. $\endgroup$ – Volka Aug 17 '17 at 3:28
  • $\begingroup$ Is it exactly latent dirichlet allocation, because that becomes important. $\endgroup$ – Thomas Cleberg Aug 17 '17 at 3:29
6
$\begingroup$

To compare two LDA topics, you're really trying to compute the distance between two probability distributions.

One such measure that's commonly used in these circumstances is the Hellinger Distance. To find the closest match for $x_1$ in the topics for $y$, you would calulate the Hellinger Distance between $x_1$ and each $y$ topic, then take the lowest one.

Keep in mind that there's no guarantee whatsoever that the "most similar" topic in this sense would be remotely, subjectively similar.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Is there any sklearn library for Hellinger Distance? $\endgroup$ – Volka Aug 17 '17 at 5:36
  • 1
    $\begingroup$ No, but there are many ways to implement it. A few of them are discussed here: gist.github.com/larsmans/3116927 $\endgroup$ – Thomas Cleberg Aug 17 '17 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.