In one of the lecture on language modeling about calculating the perplexity of a model by Dan Jurafsky in his course on Natural Language Processing, in slide number 33 he give the formula for perplexity as

enter image description here

Then, in the next slide number 34, he presents a following scenario:

"If a system has to recognize • Operator (1 in 4) • Sales (1 in 4) • Technical Support (1 in 4) • 30,000 names (1 in 120,000 each)"

Perplexity in this case is 53.

Can anyone explain how the answer 53 came ?

  • $\begingroup$ Could you please share the code for perplexity in python as to how to compare 2 models in text generation task $\endgroup$ – Sunny Apr 24 '20 at 2:03

I believe he meant: you need to identify/predict a sequence of 4 consecutive things. First: an operator, then a sales person, then a technical support person, and finally one name out of 30,000 names. One and only one sequence is correct.

The probability of the correct sequence:

$$(1/4)*(1/4)*(1/4)*(1/120,000) = 0.0000001302083333$$

If you get the 4th root, that gives you the geometric mean (in some sense that's the average per step for four steps)

$$(0.0000001302083333)^.25 = 0.01899589214 ≈ (1/53)$$


$$(1/53)*(1/53)*(1/53)*(1/53) ≈ (1/4)*(1/4)*(1/4)*(1/120,000)$$

It was however, not clear in the slides or the explanation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.