I'm using java mallet's LDA engine to determine topics for a set of (some 30K) documents.
I've managed to train the model and serialize/deserialize it as needed.
The question is how do I find the top 'n' documents in each topic.
The closest thing I can find is 'getTopicDocuments' but this returns an array of integer ids and I have no idea what they relate to.
For the record he is my code I use to train the documents:
static void buildModel() throws IOException {
// Begin by importing documents from text to feature sequences
ArrayList<Pipe> pipeList = new ArrayList<Pipe>();
// Pipes: lowercase, tokenize, remove stopwords, map to features
// my own pipe that loads the the contents
// of each file into the pipeline.
pipeList.add(new FileToCharSequence());
pipeList.add(new CharSequenceLowercase());
pipeList.add(new CharSequence2TokenSequence(Pattern.compile("\\p{L}[\\p{L}\\p{P}]+\\p{L}")));
// created a snowball based stemming Pipe
pipeList.add(new Stemming());
/// my ownversion of stopwords that stemms them
pipeList.add(new TokenSequenceRemoveStemmedStopwords(
new File(
"stoplist.txt"),
"UTF-8", false, false, false));
pipeList.add(new TokenSequenceRemoveNonAlpha());
pipeList.add(new TokenSequence2FeatureSequence());
InstanceList instances = new InstanceList(new SerialPipes(pipeList));
var fileIterator = new UnlabeledFileIterator(new File("\path\to\docs"));
instances.addThruPipe(fileIterator);
System.out.println("Starting");
// Create a model with 100 topics, alpha_t = 0.01, beta_w = 0.01
// Note that the first parameter is passed as the sum over topics, while
// the second is the parameter for a single dimension of the Dirichlet prior.
int numTopics = 180;
ParallelTopicModel model = new ParallelTopicModel(numTopics, 1.0, 0.01);
model.addInstances(instances);
// Use two parallel samplers, which each look at one half the corpus and combine
// statistics after every iteration.
model.setNumThreads(8);
final int iterations = 2000;
// Run the model for 50 iterations and stop (this is for testing only,
// for real applications, use 1000 to 2000 iterations)
model.setNumIterations(iterations);
model.estimate();
model.write(new File("LDA.model"));
}