# Calculationg perplexity (in natural language processing) manually

I am trying to understand Perplexity within Natural Language Processing as a metric more fully. And I am doing so by creating manual examples to understand all the component parts. Is the following correctly understood:

Given a lists W of words (as probabilities), where W consists of $$w_1$$ .. $$w_n$$ , and where we know the probabilities for each word, a model will still have to compute the intersection of words for the following formula to be useful:

$$P(W)=P\left(w_1\right) P\left(w_2 \mid w_1\right) P\left(w_3 \mid w_2, w_1\right) \ldots P\left(w_N \mid w_{N-1}, w_{N-2}\right)$$

Since the formula for conditional probability is given by:

$$P(A \mid B)=\frac{P(A \cap B)}{P(B)}$$

And the intersection $${P(A \cap B)}$$ will in the instance of NLP be calculated by Cross Entropy loss in a model.

• Is the question understandable? Commented Nov 6, 2022 at 9:32
• My guess would be that $P(A|B)$ will be measured, and therefore forms a model, that can then come up with prediction of the kind that is the case for a logistic regression. Commented Nov 6, 2022 at 9:54
• Your last sentence is strange to me: why would the intersection $P(A\cap B)$ be calculated by cross-entropy? (and how?). It's calculated the usual way, by dividing the frequency of $A\cap B$ by the size of the sample space (in this case corpus size minus 1). Commented Nov 7, 2022 at 19:37
• Because in a text of length N there are exactly N-1 pairs of consecutive tokens (bigrams). Example: A B C D -> AB, BC, CD (4-1=3). note: it's N-2 for trigrams, N-3 for 4-grams, etc. Commented Nov 8, 2022 at 11:01
• Well I'm not so clear myself about perplexity but I know the basics of language models (LMs). Actually your formula is for trigrams, but yes, LMs represent n-grams probabilities so P(W2|W1) means 'prob of W2 knowing the prev word W1'. This is exactly how context is taken into account, by how likely a word is given a sequence of previous words. Yes, your count is correct for a sentence of 4 words. But of course one obtains more accurate probs on a corpus with millions words ;) Commented Nov 8, 2022 at 12:23