Sub topics with Latent Dirichlet Allocation

Question

I'm training an LDA model with gensim's LdaMulticore. The topics look great, but knowing the domain I know there exists topics within topics but I'm not quite sure the best way to model this.

I've come across this implementation of Hierarchical LDA, but I'm having a hard time implementing it (no community support). And I don't think gensim's hdpModel is what I want, given this discussion.

I'm currently doing this:

1) train LDA model on all records to get general topics

2) use this LDA model to assign each record a primary topic

3) for each topic, retrieve only the records that were assigned that topic

4) train a new LDA model only on the filtered records (for example, where topic ID == 3) to generate sub topics from filtered set

5) Assign each record a general topic ID and a sub topic ID

Is this a valid way to get topics and sub topics? Should I be weary of this approach?

Thanks for any insight.

Answer 1

I actually do not think your method is a good way to find subtopics. Consider a document X with a distribution of topics z. X is made up of a mixed model distribution of topic Z. If you just give a document the most domiant distribution, and then run lda again, you might find subtopics but you'll also refind the topics that should should perhaps not be considered subtopics.

For example, let's consider a document that talks about food and exercising and why certain foods and cardo is good for heart health. Let's assume that we find topics 1 and 2 such that the main topics are food, and exercise. Suppose that the domiant distribution is topic 1. Then running LDA on that particular document again, you're going to refine food, exercise and perhaps nutrition, viatims, etc. However, there's no real reasons what should be considered the topic without inspecting each one and making an inference regarding it. Also you'll have no real way to discern how deep you should go down this tree.

The original paper on hLDA can prove useful and exploits the chinese restraurant process and its relationship to dirichlet distribution to form the topic relationship and tree depth problem. Also on David Blei github (bleilab) has an implentation in c++. You can essentially setup a little shell script after processing the data in a langauge you may be more familar with and use his code.

Answer 2

After running LDA, shouldn't your result already look like: Record1-> 71% topic 1, 14% topic 2, 15% topic 3, so topic 2 and topic 3 would be subtopics of record 1?

If you want to find subtopics of topic 1, you could search all your records, where topic 1 is the general topic and look for the most common topics that appear in records where topic 1 is the general topic.