Consider a (somewhat nonsensical) sentence - "I see saw a see saw"

The observed bi-grams would be: "I see"

"see saw"

"saw a"


"a see".

My aim is to smoothen out the probability mass of the bi-gram probabilities by using Good-Turing smoothing. For this, I need to find the count of unseen bi-grams, i.e., bi-grams with a frequency count of 0.

How do I do this?

1) Would this be a list of all bi-grams formed by using 2 non-consecutive words? For example, "I saw", "saw saw", "a I", etc.?

2) Would repetitions of the same word be included as bi-grams? Eg. "I I", "see see", etc.?


2 Answers 2


I just remembered that we create a table with all possible words as the header of each row and of each column. As a result, the list of all bi-grams would be all possible bi-grams formed by concatenating any 2 words.


Bi-grams are all possible unique permutations of two tokens in a vocabulary, given a vocabulary is the set of all tokens in a corpus.

Here is Python code that calculates the count of unseen bi-grams:

from itertools import permutations

corpus = "I see saw a see saw"
tokens = corpus.split()
vocab = set(tokens)
all_bigrams = set(permutations(vocab, r=2))
observed_bigrams = {(t_1, t_2) for t_1, t_2 in zip(tokens, tokens[1:])}
count_of_unobserved_bigrams = len(all_bigrams) - len(observed_bigrams)
print(f"The count of unobserved bigrams: {count_of_unobserved_bigrams}")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.