I am using Gensim Phrases to detct n-grams in my text. Thus, I am interested in knowing the mechanism that Phrases uses to detect these n-grams in the text. Can someone please explain the mechanism used in Phrases in simple terms?
As the gensim tool cites the very famous paper by Mikolov - "Distributed Representations of Words and Phrases..." using which it is implemented. In the paper if you look at the section "4 Learning Phrases" they give a nice explanation of how n-grams are calculated (Equation 6).
So, if want to count bigrams this formula is straight-forward; score(wi, wj) is the score between any two words occuring together. But when counting trigrams, 'wi' will be a bigram and 'wj' will be a word. And same follows for any number of grams.
Gensim detects a bigram if a scoring function for two words exceeds a threshold (which is a parameter for Phrases).
The default scoring function is what is in the answer by flyingDope, but multiplied by vocabulary size (use
help(Phraser) or see the gensim's Github repository (gensim/models/phrases.py)):
def original_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count, corpus_word_count): #... """ worda_count : int Number of occurrences for first word. wordb_count : int Number of occurrences for second word. bigram_count : int Number of co-occurrences for phrase "worda_wordb". len_vocab : int Size of vocabulary. min_count: int Minimum collocation count threshold. corpus_word_count : int Not used in this particular scoring technique. """ #... return (bigram_count - min_count) / worda_count / wordb_count * len_vocab
Another implemented score function is
npmi_scorer based on a paper by G. Bouma.
I think n-grams for n>2 are done by applying bigram detection
min_count (i.e. $delta$) was zero and if instead
len_vocab we multiplied by
corpus_word_count, then the result of
original_scorer would be essentially the ratio of the probability to see
worda and the unconditional probability to see
wordb at a random position, that is how many times the presence of
worda increases the probability to see
wordb in the next position.
I cannot understand why gensim chose to use
len_vocab here, but perhaps they had some reason to. You can pass your own scoring function as well.