0
$\begingroup$

I created a MLE bigram language model on a text, however, I don't know how to apply it on test data:

The following is my try:

pp1 = model.perplexity([('users','never')])
text = "users never"
ll = list(bigrams(text.split()))
print(list(ll))
pp2 = model.perplexity('users never')
pp3 = model.perplexity(ll)
print("pp:",pp1, " pp2", pp2, " pp3", pp3)

And this is the output:

pp: 3.0000000000000004  pp2 inf  pp3 3.0000000000000004

It seems the input of perplexity should be a bigram sequence.

However, when I try to use this logic in another model trained with more data, I got the division by zero error if I use it as bigram sequence, but a number if I use the raw text. So, I'm not sure which method is correct to evaluate my model on test data, a raw text or a sequence of bigrams.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.