I am using Gensim Phrases to detct n-grams in my text. Thus, I am interested in knowing the mechanism that Phrases uses to detect these n-grams in the text. Can someone please explain the mechanism used in Phrases in simple terms?
2 Answers
As the gensim tool cites the very famous paper by Mikolov - "Distributed Representations of Words and Phrases..." using which it is implemented. In the paper if you look at the section "4 Learning Phrases" they give a nice explanation of how n-grams are calculated (Equation 6).
So, if want to count bigrams this formula is straight-forward; score(wi, wj) is the score between any two words occuring together. But when counting trigrams, 'wi' will be a bigram and 'wj' will be a word. And same follows for any number of grams.
Gensim detects a bigram if a scoring function for two words exceeds a threshold (which is a parameter for Phrases).
The default scoring function is what is in the answer by flyingDope, but multiplied by vocabulary size (use help(Phraser)
or see the gensim's Github repository (gensim/models/phrases.py)):
def original_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count, corpus_word_count):
#...
"""
worda_count : int
Number of occurrences for first word.
wordb_count : int
Number of occurrences for second word.
bigram_count : int
Number of co-occurrences for phrase "worda_wordb".
len_vocab : int
Size of vocabulary.
min_count: int
Minimum collocation count threshold.
corpus_word_count : int
Not used in this particular scoring technique.
"""
#...
return (bigram_count - min_count) / worda_count / wordb_count * len_vocab
Another implemented score function is npmi_scorer
based on a paper by G. Bouma.
I think n-grams for n>2 are done by applying bigram detection n-1
times.
If min_count
(i.e. $delta$) was zero and if instead len_vocab
we multiplied by corpus_word_count
, then the result of original_scorer
would be essentially the ratio of the probability to see wordb
following worda
and the unconditional probability to see wordb
at a random position, that is how many times the presence of worda
increases the probability to see wordb
in the next position.
I cannot understand why gensim chose to use len_vocab
here, but perhaps they had some reason to. You can pass your own scoring function as well.