# How to compute unseen bi-grams in a corpus (for Good-Turing Smoothing)

Consider a (somewhat nonsensical) sentence - "I see saw a see saw"

The observed bi-grams would be: "I see"

"see saw"

"saw a"

and,

"a see".

My aim is to smoothen out the probability mass of the bi-gram probabilities by using Good-Turing smoothing. For this, I need to find the count of unseen bi-grams, i.e., bi-grams with a frequency count of 0.

How do I do this?

1) Would this be a list of all bi-grams formed by using 2 non-consecutive words? For example, "I saw", "saw saw", "a I", etc.?

2) Would repetitions of the same word be included as bi-grams? Eg. "I I", "see see", etc.?

I just remembered that we create a table with all possible words as the header of each row and of each column. As a result, the list of all bi-grams would be all possible bi-grams formed by concatenating any 2 words.

Bi-grams are all possible unique permutations of two tokens in a vocabulary, given a vocabulary is the set of all tokens in a corpus.

Here is Python code that calculates the count of unseen bi-grams:

from itertools import permutations

corpus = "I see saw a see saw"
tokens = corpus.split()
vocab = set(tokens)
all_bigrams = set(permutations(vocab, r=2))
observed_bigrams = {(t_1, t_2) for t_1, t_2 in zip(tokens, tokens[1:])}
count_of_unobserved_bigrams = len(all_bigrams) - len(observed_bigrams)
print(f"The count of unobserved bigrams: {count_of_unobserved_bigrams}")