I created a
MLE bigram language model on a text, however, I don't know how to apply it on test data:
The following is my try:
pp1 = model.perplexity([('users','never')]) text = "users never" ll = list(bigrams(text.split())) print(list(ll)) pp2 = model.perplexity('users never') pp3 = model.perplexity(ll) print("pp:",pp1, " pp2", pp2, " pp3", pp3)
And this is the output:
pp: 3.0000000000000004 pp2 inf pp3 3.0000000000000004
It seems the input of
perplexity should be a bigram sequence.
However, when I try to use this logic in another model trained with more data, I got the
division by zero error if I use it as
bigram sequence, but a number if I use the raw text. So, I'm not sure which method is correct to evaluate my model on test data, a raw text or a sequence of bigrams.