I created a MLE bigram language model on a text, however, I don't know how to apply it on test data:

The following is my try:

pp1 = model.perplexity([('users','never')])
text = "users never"
ll = list(bigrams(text.split()))
pp2 = model.perplexity('users never')
pp3 = model.perplexity(ll)
print("pp:",pp1, " pp2", pp2, " pp3", pp3)

And this is the output:

pp: 3.0000000000000004  pp2 inf  pp3 3.0000000000000004

It seems the input of perplexity should be a bigram sequence.

However, when I try to use this logic in another model trained with more data, I got the division by zero error if I use it as bigram sequence, but a number if I use the raw text. So, I'm not sure which method is correct to evaluate my model on test data, a raw text or a sequence of bigrams.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.