I'm prototyping an application and I need a language model to compute perplexity on some generated sentences.
Is there any trained language model in python I can readily use? Something simple like
model = LanguageModel('en') p1 = model.perplexity('This is a well constructed sentence') p2 = model.perplexity('Bunny lamp robert junior pancake') assert p1 < p2
I've looked at some frameworks but couldn't find what I want. I know I can use something like:
from nltk.model.ngram import NgramModel lm = NgramModel(3, brown.words(categories='news'))
This uses a good turing probability distribution on Brown Corpus, but I was looking for some well-crafted model on some big dataset, like the 1b words dataset. Something that I can actually trust the results for a general domain (not only news)