Skip to main content
added 51 characters in body
Source Link
Valentas
  • 1.3k
  • 1
  • 9
  • 21

Gensim detects a bigram if a scoring function for two words exceeds a threshold (which is a parameter for Phrases).

The default scoring function is what is in the answer by flyingDope, but multiplied by vocabulary size (use help(Phraser) or see the gensim's Github repository (gensim/models/phrases.py)):

def original_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count, corpus_word_count):
   #...
   """
    worda_count : int
        Number of occurrences for first word.
    wordb_count : int
        Number of occurrences for second word.
    bigram_count : int
        Number of co-occurrences for phrase "worda_wordb".
    len_vocab : int
        Size of vocabulary.
    min_count: int
        Minimum collocation count threshold.
    corpus_word_count : int
        Not used in this particular scoring technique.
    """
    #...

    return (bigram_count - min_count) / worda_count / wordb_count * len_vocab

Another implemented score function is npmi_scorer based on a paper by G. Bouma.

I think n-grams for n>2 are done by applying bigram detection n-1 times.

If min_count (i.e. $delta$) was zero and if instead len_vocab we multiplied by corpus_word_count, then the result of original_scorer would be essentially the ratio of the probability to see wordb following worda and the unconditional probability to see wordb at a random position, that is how many times the presence of worda increases the probability to see wordb in the next position. 

I cannot understand why gensim chose to use word_countlen_vocab here, but perhaps they had some reason to. You can pass your own scoring function as well.

Gensim detects a bigram if a scoring function for two words exceeds a threshold (which is a parameter for Phrases).

The default scoring function is what is in the answer by flyingDope, but multiplied by vocabulary size (use help(Phraser) or see the gensim's Github repository (gensim/models/phrases.py)):

def original_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count, corpus_word_count):
   #...
   """
    worda_count : int
        Number of occurrences for first word.
    wordb_count : int
        Number of occurrences for second word.
    bigram_count : int
        Number of co-occurrences for phrase "worda_wordb".
    len_vocab : int
        Size of vocabulary.
    min_count: int
        Minimum collocation count threshold.
    corpus_word_count : int
        Not used in this particular scoring technique.
    """
    #...

    return (bigram_count - min_count) / worda_count / wordb_count * len_vocab

Another implemented score function is npmi_scorer based on a paper by G. Bouma.

I think n-grams for n>2 are done by applying bigram detection n-1 times.

If min_count (i.e. $delta$) was zero and if instead len_vocab we multiplied by corpus_word_count, then the result of original_scorer would be essentially the ratio of the probability to see wordb following worda and the unconditional probability to see wordb at a random position, that is how many times the presence of worda increases the probability to see wordb in the next position. I cannot understand why gensim chose to use word_count here, but perhaps they had some reason to.

Gensim detects a bigram if a scoring function for two words exceeds a threshold (which is a parameter for Phrases).

The default scoring function is what is in the answer by flyingDope, but multiplied by vocabulary size (use help(Phraser) or see the gensim's Github repository (gensim/models/phrases.py)):

def original_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count, corpus_word_count):
   #...
   """
    worda_count : int
        Number of occurrences for first word.
    wordb_count : int
        Number of occurrences for second word.
    bigram_count : int
        Number of co-occurrences for phrase "worda_wordb".
    len_vocab : int
        Size of vocabulary.
    min_count: int
        Minimum collocation count threshold.
    corpus_word_count : int
        Not used in this particular scoring technique.
    """
    #...

    return (bigram_count - min_count) / worda_count / wordb_count * len_vocab

Another implemented score function is npmi_scorer based on a paper by G. Bouma.

I think n-grams for n>2 are done by applying bigram detection n-1 times.

If min_count (i.e. $delta$) was zero and if instead len_vocab we multiplied by corpus_word_count, then the result of original_scorer would be essentially the ratio of the probability to see wordb following worda and the unconditional probability to see wordb at a random position, that is how many times the presence of worda increases the probability to see wordb in the next position. 

I cannot understand why gensim chose to use len_vocab here, but perhaps they had some reason to. You can pass your own scoring function as well.

added 110 characters in body
Source Link
Valentas
  • 1.3k
  • 1
  • 9
  • 21

Gensim detects a bigram if a scoring function for two words exceeds a threshold (which is a parameter for Phrases).

The default scoring function is what is in the answer by flyingDope, but multiplied by vocabulary size (use help(Phraser) or see the gensim's Github repository (gensim/models/phrases.py)):

def original_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count, corpus_word_count):
   #...
   """
    worda_count : int
        Number of occurrences for first word.
    wordb_count : int
        Number of occurrences for second word.
    bigram_count : int
        Number of co-occurrences for phrase "worda_wordb".
    len_vocab : int
        Size of vocabulary.
    min_count: int
        Minimum collocation count threshold.
    corpus_word_count : int
        Not used in this particular scoring technique.
    """
    #...

    return (bigram_count - min_count) / worda_count / wordb_count * len_vocab

Another implemented score function is npmi_scorer based on a paper by G. Bouma.

I think n-grams for n>2 are done by applying bigram detection n-1 times.

If min_count (i.e. $delta$) was zero and if instead len_vocab we multiplied by corpus_word_count, then the result of original_scorer would be essentially the ratio of the probability to see wordb following worda and the unconditional probability to see wordb at a random position, that is how many times the presence of worda increases the probability to see wordb in the next position. I cannot understand why gensim chose to use word_count here, but perhaps they had some reason to.

Gensim detects a bigram if a scoring function for two words exceeds a threshold (which is a parameter for Phrases).

The default scoring function is what is in the answer by flyingDope, but multiplied by vocabulary size (use help(Phraser) or see the gensim's Github repository (gensim/models/phrases.py)):

def original_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count, corpus_word_count):
   #...
   """
    worda_count : int
        Number of occurrences for first word.
    wordb_count : int
        Number of occurrences for second word.
    bigram_count : int
        Number of co-occurrences for phrase "worda_wordb".
    len_vocab : int
        Size of vocabulary.
    min_count: int
        Minimum collocation count threshold.
    corpus_word_count : int
        Not used in this particular scoring technique.
    """
    #...

    return (bigram_count - min_count) / worda_count / wordb_count * len_vocab

Another implemented score function is npmi_scorer based on a paper by G. Bouma.

I think n-grams for n>2 are done by applying bigram detection n-1 times.

If min_count (i.e. $delta$) was zero and if instead len_vocab we multiplied by corpus_word_count, then the result of original_scorer would be essentially the ratio of the probability to see wordb following worda and the unconditional probability to see wordb at a random position. I cannot understand why gensim chose to use word_count here, but perhaps they had some reason to.

Gensim detects a bigram if a scoring function for two words exceeds a threshold (which is a parameter for Phrases).

The default scoring function is what is in the answer by flyingDope, but multiplied by vocabulary size (use help(Phraser) or see the gensim's Github repository (gensim/models/phrases.py)):

def original_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count, corpus_word_count):
   #...
   """
    worda_count : int
        Number of occurrences for first word.
    wordb_count : int
        Number of occurrences for second word.
    bigram_count : int
        Number of co-occurrences for phrase "worda_wordb".
    len_vocab : int
        Size of vocabulary.
    min_count: int
        Minimum collocation count threshold.
    corpus_word_count : int
        Not used in this particular scoring technique.
    """
    #...

    return (bigram_count - min_count) / worda_count / wordb_count * len_vocab

Another implemented score function is npmi_scorer based on a paper by G. Bouma.

I think n-grams for n>2 are done by applying bigram detection n-1 times.

If min_count (i.e. $delta$) was zero and if instead len_vocab we multiplied by corpus_word_count, then the result of original_scorer would be essentially the ratio of the probability to see wordb following worda and the unconditional probability to see wordb at a random position, that is how many times the presence of worda increases the probability to see wordb in the next position. I cannot understand why gensim chose to use word_count here, but perhaps they had some reason to.

Source Link
Valentas
  • 1.3k
  • 1
  • 9
  • 21

Gensim detects a bigram if a scoring function for two words exceeds a threshold (which is a parameter for Phrases).

The default scoring function is what is in the answer by flyingDope, but multiplied by vocabulary size (use help(Phraser) or see the gensim's Github repository (gensim/models/phrases.py)):

def original_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count, corpus_word_count):
   #...
   """
    worda_count : int
        Number of occurrences for first word.
    wordb_count : int
        Number of occurrences for second word.
    bigram_count : int
        Number of co-occurrences for phrase "worda_wordb".
    len_vocab : int
        Size of vocabulary.
    min_count: int
        Minimum collocation count threshold.
    corpus_word_count : int
        Not used in this particular scoring technique.
    """
    #...

    return (bigram_count - min_count) / worda_count / wordb_count * len_vocab

Another implemented score function is npmi_scorer based on a paper by G. Bouma.

I think n-grams for n>2 are done by applying bigram detection n-1 times.

If min_count (i.e. $delta$) was zero and if instead len_vocab we multiplied by corpus_word_count, then the result of original_scorer would be essentially the ratio of the probability to see wordb following worda and the unconditional probability to see wordb at a random position. I cannot understand why gensim chose to use word_count here, but perhaps they had some reason to.