5
$\begingroup$

I have two sets of topics obtained from two different sets of news paper articles.

In other words, Cluster_1 = ${x_1, x_2, ..., x_n}$ includes the main topics of 'X' news paper set and Cluster_2 = ${y_1, y_2, ..., y_n}$ includes the main topics of 'Y' news paper set.

Now I want to find clusters in the two sets that are similar/related by considering the cluster attributes as given in the example below.

Example 1,
**X1 in Cluster_1** is mostly similar/related to **Y2 in Cluster_2**
**X2 in Cluster_1** is mostly similar/related to **Yn in cluster_2**
and so on.

Example 2:
News about Yet in Cluster_1 is mostly similar/related to News about Science in Cluster_2
News about Floods in Cluster_1 is mostly similar/related to News about Rains in Cluster_2

Since, I am dealing with two separate sets of clusters, what would be a suitable measurement/method I can use to connect the clusters in the two different sets?

Improve this question
$\endgroup$
6
  • $\begingroup$ Are these two different groupings of the same observations? In the same space? Same dimensional space? What does "most similar/related" mean to you in this context? $\endgroup$
    – Thomas Cleberg
    Aug 17, 2017 at 2:31
  • $\begingroup$ Thank you for the comment. I edited the question by including the missing information you have mentioned. $\endgroup$
    – Volka
    Aug 17, 2017 at 2:48
  • $\begingroup$ Are your clusters the result of topic modeling with something like Latent dirichlet allocation over two newspapers, and you're wondering if you can compare the topics between the two newspapers? $\endgroup$
    – Thomas Cleberg
    Aug 17, 2017 at 3:26
  • $\begingroup$ Yes, you are correct. $\endgroup$
    – Volka
    Aug 17, 2017 at 3:28
  • $\begingroup$ Is it exactly latent dirichlet allocation, because that becomes important. $\endgroup$
    – Thomas Cleberg
    Aug 17, 2017 at 3:29

1 Answer 1

Reset to default
6
$\begingroup$

To compare two LDA topics, you're really trying to compute the distance between two probability distributions.

One such measure that's commonly used in these circumstances is the Hellinger Distance. To find the closest match for $x_1$ in the topics for $y$, you would calulate the Hellinger Distance between $x_1$ and each $y$ topic, then take the lowest one.

Keep in mind that there's no guarantee whatsoever that the "most similar" topic in this sense would be remotely, subjectively similar.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Is there any sklearn library for Hellinger Distance? $\endgroup$
    – Volka
    Aug 17, 2017 at 5:36
  • 1
    $\begingroup$ No, but there are many ways to implement it. A few of them are discussed here: gist.github.com/larsmans/3116927 $\endgroup$
    – Thomas Cleberg
    Aug 17, 2017 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.