I'm training an LDA model with gensim's LdaMulticore. The topics look great, but knowing the domain I know there exists topics within topics but I'm not quite sure the best way to model this.
I've come across this implementation of Hierarchical LDA, but I'm having a hard time implementing it (no community support). And I don't think gensim's hdpModel is what I want, given this discussion.
I'm currently doing this:
1) train LDA model on all records to get general topics
2) use this LDA model to assign each record a primary topic
3) for each topic, retrieve only the records that were assigned that topic
4) train a new LDA model only on the filtered records (for example, where topic ID == 3) to generate sub topics from filtered set
5) Assign each record a general topic ID and a sub topic ID
Is this a valid way to get topics and sub topics? Should I be weary of this approach?
Thanks for any insight.