# Compare two topic modelling sets

I have two sets of newspaper articles where I train the first newspaper dataset separately to get the topics per each newspaper article.

E.g., first newspaper dataset
article_1 = {'politics': 0.1, 'nature': 0.8, ..., 'sports':0, 'wild-life':1}


Again, I train my second newspaper dataset (from a different distributor) to get the topics per each newspaper article.

E.g., second newspaper dataset (from a different distributor)
article_2 = {'people': 0.3, 'animals': 0.7, ...., 'business':0.7, 'sports':0.2}


As shown in the examples, the topics I get from the two datasets are different, thus I manually matched similar topics based on their frequent words.

I want to identify whether the two newspaper distributors publish the same news in every week.

Hence, I am interested in knowing if there is a systematic way of comparing the topics across two corpora and measuring their similarity. Please help me.

• Interesting question. what is the technique you used for topic-modeling? Oct 13, 2017 at 1:07
• I would find a pooled topic model, then compare the individual distributions (e.g., by KLD).
– Emre
Oct 16, 2017 at 1:07
• I mean find the topics assuming all the articles come from a common source. KLD is en.wikipedia.org/wiki/Kullback–Leibler_divergence
– Emre
Oct 16, 2017 at 1:44
• I would use a hierchical model that defines the newspaper Dirichlet distributions based on the pooled topic model. You should be able to implement this in Edward; see this GitHub discussion. Once you have the newspaper distributions, the difference can be defined by the KLD or the symmetric Jensen-Shannon divergence as follows: bariskurt.com/… Good luck!
– Emre
Oct 17, 2017 at 4:30
• This lecture might help. Read about LDA, Dirichlet distribution, and divergence measures.
– Emre
Oct 17, 2017 at 23:59