0
$\begingroup$

I'm using java mallet's LDA engine to determine topics for a set of (some 30K) documents.

I've managed to train the model and serialize/deserialize it as needed.

The question is how do I find the top 'n' documents in each topic.

The closest thing I can find is 'getTopicDocuments' but this returns an array of integer ids and I have no idea what they relate to.

For the record he is my code I use to train the documents:

static void buildModel() throws IOException {

        // Begin by importing documents from text to feature sequences
        ArrayList<Pipe> pipeList = new ArrayList<Pipe>();

        // Pipes: lowercase, tokenize, remove stopwords, map to features
        // my own pipe that loads the the contents
        // of each file into the pipeline.
        pipeList.add(new FileToCharSequence());
        pipeList.add(new CharSequenceLowercase());
        pipeList.add(new CharSequence2TokenSequence(Pattern.compile("\\p{L}[\\p{L}\\p{P}]+\\p{L}")));
        // created a snowball based stemming Pipe
        pipeList.add(new Stemming());
       /// my ownversion of stopwords that stemms them
        pipeList.add(new TokenSequenceRemoveStemmedStopwords(
                new File(
                        "stoplist.txt"),
                "UTF-8", false, false, false));
        pipeList.add(new TokenSequenceRemoveNonAlpha());
        pipeList.add(new TokenSequence2FeatureSequence());

        InstanceList instances = new InstanceList(new SerialPipes(pipeList));


        var fileIterator = new UnlabeledFileIterator(new File("\path\to\docs"));

        
        instances.addThruPipe(fileIterator);
        System.out.println("Starting");


        // Create a model with 100 topics, alpha_t = 0.01, beta_w = 0.01
        // Note that the first parameter is passed as the sum over topics, while
        // the second is the parameter for a single dimension of the Dirichlet prior.
        int numTopics = 180;
        ParallelTopicModel model = new ParallelTopicModel(numTopics, 1.0, 0.01);

        model.addInstances(instances);

        // Use two parallel samplers, which each look at one half the corpus and combine
        // statistics after every iteration.
        model.setNumThreads(8);

        final int iterations = 2000;

        // Run the model for 50 iterations and stop (this is for testing only,
        // for real applications, use 1000 to 2000 iterations)
        model.setNumIterations(iterations);
        model.estimate();

        model.write(new File("LDA.model"));

    }
$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.