I'm prototyping an application and I need a language model to compute perplexity on some generated sentences.

Is there any trained language model in python I can readily use? Something simple like

model = LanguageModel('en')
p1 = model.perplexity('This is a well constructed sentence')
p2 = model.perplexity('Bunny lamp robert junior pancake')
assert p1 < p2

I've looked at some frameworks but couldn't find what I want. I know I can use something like:

from nltk.model.ngram import NgramModel
lm = NgramModel(3, brown.words(categories='news'))

This uses a good turing probability distribution on Brown Corpus, but I was looking for some well-crafted model on some big dataset, like the 1b words dataset. Something that I can actually trust the results for a general domain (not only news)

  • 2
    $\begingroup$ Tensorflow 1b words LM $\endgroup$ – user12075 Sep 20 '18 at 15:46
  • $\begingroup$ Well this is not at all readily usable but it's something. Thanks :) $\endgroup$ – Fred Sep 20 '18 at 17:57
  • $\begingroup$ That's a pre-trained model that you can simply download and run, and you think that is "not at all readily usable" ... $\endgroup$ – user12075 Sep 20 '18 at 18:06
  • 1
    $\begingroup$ I think you and me have very different definitions of what "readily usable" means... I would need to figure out how to get the tensorflow ops I want (input and output) and how they behave, figure out if there's any preprocessing to this and then wrap everything in some perplexity function. I'm not saying I can't do it, I'm just saying it is not at all the "readily usable" function I showed. But again, thanks for the pointer $\endgroup$ – Fred Sep 20 '18 at 18:17
  • $\begingroup$ Have you tried google? I hear they get a fair amount of data :) Not sure if they have the exact metrics you're after. cloud.google.com/natural-language/docs $\endgroup$ – flyingmeatball Sep 24 '18 at 19:04

The spaCy package has many language models, including ones trained on Common Crawl.

Language model has a specific meaning in Natural Language Processing (NlP). A language model is a probability distribution over sequences of tokens. Given a specific sequence of tokens, the model can assign a probability of that sequence appearing. SpaCy's language models include more than just a probability distribution.

The spaCy package needs to be installed and the language models need to be download:

$ pip install spacy 
$ python -m spacy download en

Then the language models can used with a couple lines of Python:

>>> import spacy
>>> nlp = spacy.load('en')

For a given model and token, there is a smoothed log probability estimate of a token's word type can be found with: token.prob attribute.

| improve this answer | |
  • $\begingroup$ Deleted my previous comments... Apparently spacy does include a proper language model (using the token.prob attribute), but it's only built in the large model version. If you edit your answer to include that info I can give you the bounty. Funny enough, I've been using spacy for months now and nowhere I saw that it had this feature $\endgroup$ – Fred Sep 28 '18 at 16:27
  • $\begingroup$ 👍 Glad you found something that works for you. $\endgroup$ – Brian Spiering Sep 29 '18 at 23:04
  • $\begingroup$ Again.. This only works if you download the large English model $\endgroup$ – Fred Sep 30 '18 at 20:19

I also think that the first answer is incorrect for the reasons that @noob333 explained.

But also Bert cannot be used out of the box as a language model. Bert gives you the p(word|context(both left and right) ) and what you want is to compute p(word|previous tokens(only left contex)). The author explains here why you cannot use it as a lm.

However you can adapt Bert and use it as a language model, as explained here.

But you can use the open ai gpt or gpt-2 pre-tained models from the same repo

Here is how you can compute the perplexity using the gpt model.

import math
from pytorch_pretrained_bert import OpenAIGPTTokenizer, OpenAIGPTModel, OpenAIGPTLMHeadModel
# Load pre-trained model (weights)
model = OpenAIGPTLMHeadModel.from_pretrained('openai-gpt')
# Load pre-trained model tokenizer (vocabulary)
tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')

def score(sentence):
    tokenize_input = tokenizer.tokenize(sentence)
    tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
    loss=model(tensor_input, lm_labels=tensor_input)
    return math.exp(loss)

a=['there is a book on the desk',
                'there is a plane on the desk',
                        'there is a book in the desk']
print([score(i) for i in a])
21.31652459381952, 61.45907380241148, 26.24923942649312
| improve this answer | |
  • $\begingroup$ its interesting how the score dropped for "in" and "on" by 5 Thinking emo $\endgroup$ – Syenix Aug 6 at 21:07

I think the accepted answer is incorrect.

token.prob is the log-prob of the token being a particular type . I am guessing 'type' refers to something like POS-tag or type of named entity (it's not clear from spacy's documentation) and the score is a confidence measure over space of all types.

This is not the same as the probabilities assigned by a language model. A language model gives you the probability distribution over all possible tokens (not the type) saying which of them is most likely to occur next.

This repo has pretty nice documentation on using BERT (a state-of-the art model) with pre-trained weights for the neural network,

I think the API's don't give you perplexity directly but you should be able to get probability scores for each token quite easily.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.