Context of question:
I want to find semantically similar documents in corpora. For that, I'm first trying Latent Dirichlet Allocation (LDA) with divergences (Hellinger, Kullback-Leibler, Jensen-Shannon) on the per document topic distributions. However, to find # of topics for my corpus ( a 948 document dataset, extracted from larger collection, where docs about the same story are humanly annotated), it was suggested to use HDP. Unfortunately, after MANY tries, I'm still unsure I'm using the packages correctly.
1) HDPFASTER continually increases the number of topics (in train.log file). What is the stopping criterion (e.g. difference in avg. likelihood between successive iterations smaller than x? which x?) and/or how many cycles should I let it run? Afterwords, do I take as the correct # of topics the last line of "train.log" OR should I also test for minimum perplexity (It has also been suggested that Bayesian Information Criterion (BIC) might be a better measure) and choose the smallest? If the latter, should I test for ALL iterations?! Is HDPFASTER's test feature the appropriate tool for this? Last, should I --sample_hyper? Do I just use a(alpha) as input to LDA's prior alpha, perhaps divided by a number (e.g. # of topics)? What about LDA's prior beta?
2) HCA continually decreases the number of topics (in *.log file). When to stop? Do I accept the final # of topics in .log ( exp.ent =number0) OR do I search for the minimum perplexity of test set (as I assume it appears as number2 in "log_2(perp)=number1,number2" in .log throughout iterations), which ALWAYS appears VERY close to my initial max number of topics? Hyperparameters alpha,beta: do I sample using -D , -E? Which do I put as input to LDA's priors? Is it generally worth it?
Generally: Is it possible that my dataset is just too small and/or 'biased/fragmented/incomplete', in the sense that each story has only 1-3 examples, while the diversity of the topics these stories discuss can be quite large? Should I just try and augment it with a larger 'homogeneous' dataset?