I am using Gensim Phrases to detct n-grams in my text. Thus, I am interested in knowing the mechanism that Phrases uses to detect these n-grams in the text. Can someone please explain the mechanism used in Phrases in simple terms?


As the gensim tool cites the very famous paper by Mikolov - "Distributed Representations of Words and Phrases..." using which it is implemented. In the paper if you look at the section "4 Learning Phrases" they give a nice explanation of how n-grams are calculated (Equation 6).

enter image description here

So, if want to count bigrams this formula is straight-forward; score(wi, wj) is the score between any two words occuring together. But when counting trigrams, 'wi' will be a bigram and 'wj' will be a word. And same follows for any number of grams.


Gensim detects a bigram if a scoring function for two words exceeds a threshold (which is a parameter for Phrases).

The default scoring function is what is in the answer by flyingDope, but multiplied by vocabulary size (use help(Phraser) or see the gensim's Github repository (gensim/models/phrases.py)):

def original_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count, corpus_word_count):
    worda_count : int
        Number of occurrences for first word.
    wordb_count : int
        Number of occurrences for second word.
    bigram_count : int
        Number of co-occurrences for phrase "worda_wordb".
    len_vocab : int
        Size of vocabulary.
    min_count: int
        Minimum collocation count threshold.
    corpus_word_count : int
        Not used in this particular scoring technique.

    return (bigram_count - min_count) / worda_count / wordb_count * len_vocab

Another implemented score function is npmi_scorer based on a paper by G. Bouma.

I think n-grams for n>2 are done by applying bigram detection n-1 times.

If min_count (i.e. $delta$) was zero and if instead len_vocab we multiplied by corpus_word_count, then the result of original_scorer would be essentially the ratio of the probability to see wordb following worda and the unconditional probability to see wordb at a random position, that is how many times the presence of worda increases the probability to see wordb in the next position.

I cannot understand why gensim chose to use len_vocab here, but perhaps they had some reason to. You can pass your own scoring function as well.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.