I am trying to understand Perplexity within Natural Language Processing as a metric more fully. And I am doing so by creating manual examples to understand all the component parts. Is the following correctly understood:

Given a lists W of words (as probabilities), where W consists of $w_1$ .. $w_n$ , and where we know the probabilities for each word, a model will still have to compute the intersection of words for the following formula to be useful:

$$ P(W)=P\left(w_1\right) P\left(w_2 \mid w_1\right) P\left(w_3 \mid w_2, w_1\right) \ldots P\left(w_N \mid w_{N-1}, w_{N-2}\right) $$

Since the formula for conditional probability is given by:

$P(A \mid B)=\frac{P(A \cap B)}{P(B)}$

And the intersection ${P(A \cap B)}$ will in the instance of NLP be calculated by Cross Entropy loss in a model.

  • $\begingroup$ Is the question understandable? $\endgroup$
    – Piskator
    Commented Nov 6, 2022 at 9:32
  • $\begingroup$ My guess would be that $P(A|B)$ will be measured, and therefore forms a model, that can then come up with prediction of the kind that is the case for a logistic regression. $\endgroup$
    – Piskator
    Commented Nov 6, 2022 at 9:54
  • $\begingroup$ Your last sentence is strange to me: why would the intersection $P(A\cap B)$ be calculated by cross-entropy? (and how?). It's calculated the usual way, by dividing the frequency of $A\cap B$ by the size of the sample space (in this case corpus size minus 1). $\endgroup$
    – Erwan
    Commented Nov 7, 2022 at 19:37
  • 1
    $\begingroup$ Because in a text of length N there are exactly N-1 pairs of consecutive tokens (bigrams). Example: A B C D -> AB, BC, CD (4-1=3). note: it's N-2 for trigrams, N-3 for 4-grams, etc. $\endgroup$
    – Erwan
    Commented Nov 8, 2022 at 11:01
  • 1
    $\begingroup$ Well I'm not so clear myself about perplexity but I know the basics of language models (LMs). Actually your formula is for trigrams, but yes, LMs represent n-grams probabilities so P(W2|W1) means 'prob of W2 knowing the prev word W1'. This is exactly how context is taken into account, by how likely a word is given a sequence of previous words. Yes, your count is correct for a sentence of 4 words. But of course one obtains more accurate probs on a corpus with millions words ;) $\endgroup$
    – Erwan
    Commented Nov 8, 2022 at 12:23


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.