Pointwise Mutual Information or PMI for short is given as


Which is the same as:


Where BigramOccurrences is number of times bigram appears as feature, 1stWordOccurrences is number of times 1st word in bigram appears as feature and 2ndWordOccurrences is number of times 2nd word from the bigram appears as feature. Finally N is given as number of total words.

We can tweak the following formula a bit and get the following:


Now the part that confuses me a bit is the N in the formula. From what I understand it should be a total number of feature occurrences, even though it is described as total number of words. So essentially I wouldn't count total number of words in dataset (as that after some preprocessing doesn't seem like it makes sense to me), but rather I should count the total number of times all bigrams that are features have appeared as well as single words, is this correct?

Finally, one other thing that confuses me a bit is when I work with more than bigrams, so for example trigrams are also part of features. I would then, when calculating PMI for a specific bigram, not consider count of trigrams for N in the given formula? Vice-versa when calculating PMI for a single trigram, the N wouldn't account for number of bigrams, is this correct?

If I misunderstood something about formula, please let me know, as the resources I found online don't make it really clear to me.


1 Answer 1


The application of PMI to text is not so straightforward, there can be different methods.

PMI is originally defined for a standard sample space of joint events, i.e. a set of instances which are either A and B, A and not B, not A and B or not A and not B. In this setting $N$ is the size of the space, of course.

So the question when dealing with text is: what is the sample space?

  • Sometimes it makes sense to consider specific units of text as instances, for example small documents (e.g. tweets) or sentences. In this option the different cases are whether word A and B appear at least once individually/jointly in the document, and then we count the number of documents as frequency. $N$ is the total number of documents, of course.
  • Sometimes there's no natural unit to consider, only the full text. In this case the sample space is defined by a moving windows of length $m$ in the text, i.e. the window starting at position 1, 2, 3, etc. Every window is a 'document' which can have combination of [not] A/B.
  • $\begingroup$ I see. Thank you very much for your reply! $\endgroup$ Jul 9, 2022 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.