0
$\begingroup$

I have a text which has many sentences. How can I use nltk.ngrams to process it?

This is my code:

   sequence = nltk.tokenize.word_tokenize(raw) 
   bigram = ngrams(sequence,2)
   freq_dist = nltk.FreqDist(bigram)
   prob_dist = nltk.MLEProbDist(freq_dist)
   number_of_bigrams = freq_dist.N()

However, the above code supposes that all sentences are one sequence. But, sentences are separated, and I guess the last word of one sentence is unrelated to the start word of another sentence. How can I create a bigram for such a text? I need also prob_dist and number_of_bigrams which are based on the `freq_dist.

There are similar questions like this https://stackoverflow.com/questions/12821201/what-are-ngram-counts-and-how-to-implement-using-nltk?rq=1 but they are mostly about a sequence of words.

$\endgroup$
3
$\begingroup$

One way is to loop through a list of sentences. Process each one sentence separately and collect the results

Brian is totally right about the solution but I think that's actually the part that is missing from his answer :) Brian's code assumes that the sentences have already been segmented, which was Ahmad's original question as far as I can tell.

With that in mind, here's an updated proposal for adding the sentence splitting to the script:

import itertools, nltk

def ngrams_wrapper(sent):
    return list(nltk.ngrams(sent, 2))

raw = "To Sherlock Holmes she is always the woman." + \
      "I have seldom heard him mention her under any other name."

# Given a string as input (`raw`), split it into sentences.
# Returns an object of type 'list: str'
sentences = nltk.sent_tokenize(raw)
# ['To Sherlock Holmes she is always the woman.',
# 'I have seldom heard him mention her under any other name.']

# Next, tokenize every sentence (string) in the list of sentences. The tokenizer takes
# strings as input so we need to apply it on each element of `sentences` (we can't apply
# it on the list itself). For that, we can use the function `map`, which applies any
# callable Python object to every element of a list.
# The output of this step will be an object of type
#   'list: list: str'
# such that
#   'sentences: sentence: word'
tokenized = map(nltk.tokenize.word_tokenize, sentences)
# [['To', 'Sherlock', 'Holmes', 'she', 'is', 'always', 'the', 'woman', '.'],
# ['I', 'have', 'seldom', 'heard', 'him', 'mention', 'her', 'under', 'any', 'other',
# 'name', '.']]

# We now generate the ngrams for each tokenized sentence in the list of sentences. Since
# we want to generate the ngrams for each sentence specifically, we again must apply the
# function to each element of the list separately, not the whole list. So, we use the
# `map` function again.
# The output of this step will be an object of type:
#   'list: list: tuple: <str, str>'
# such that
#   'sentences: sentence: ngram: <word_1, word_2>
bigrams = map(ngrams_wrapper, tokenized)
# [[('To', 'Sherlock'), ('Sherlock', 'Holmes'), ('Holmes', 'she'), ('she', 'is'), ('is',
# 'always'), ('always', 'the'), ('the', 'woman'), ('woman', '.')], [('I', 'have'),
# ('have', 'seldom'), ('seldom', 'heard'), ('heard', 'him'), ('him', 'mention'),
# ('mention', 'her'), ('her', 'under'), ('under', 'any'), ('any', 'other'), ('other',
# 'name'), ('name', '.')]]

# Finally, since we want all the bigrams in a single list to work with them more easily,
# we flatten the list, that is, we concatenate all the elements at the first level of
# depth in the list. So, if the list was an object of type
#   'list: list: tuple: <str, str>'  (sentences: sentence: ngram)
# we will transform it into:
#   'list: tuple: <str, str>'        (sentences: ngram)
bigram = list(itertools.chain.from_iterable(bigrams))
# [('To', 'Sherlock'), ('Sherlock', 'Holmes'), ('Holmes', 'she'), ('she', 'is'), ('is',
# 'always'), ('always', 'the'), ('the', 'woman'), ('woman', '.'), ('I', 'have'), ('have',
# 'seldom'), ('seldom', 'heard'), ('heard', 'him'), ('him', 'mention'), ('mention',
# 'her'), ('her', 'under'), ('under', 'any'), ('any', 'other'), ('other', 'name'),
# ('name', '.')]

freq_dist = nltk.FreqDist(bigram)
# [(('always', 'the'), 1), (('woman', '.'), 1), (('Holmes', 'she'), 1), (('seldom',
# 'heard'), 1), (('Sherlock', 'Holmes'), 1), (('him', 'mention'), 1), (('I', 'have'), 1),
# (('any', 'other'), 1), (('under', 'any'), 1), (('the', 'woman'), 1), (('her', 'under'),
# 1), (('other', 'name'), 1), (('To', 'Sherlock'), 1), (('name', '.'), 1), (('she', 'is'),
# 1), (('heard', 'him'), 1), (('mention', 'her'), 1), (('have', 'seldom'), 1), (('is',
# 'always'), 1)]

prob_dist = nltk.MLEProbDist(freq_dist)
number_of_bigrams = freq_dist.N()
$\endgroup$
1
$\begingroup$

One way is to loop through a list of sentences. Process each one sentence separately and collect the results:

import nltk
from nltk.tokenize import word_tokenize
from nltk.util import ngrams 

sentences = ["To Sherlock Holmes she is always the woman.", "I have seldom heard him mention her under any other name."]

bigrams = []
for sentence in sentences:
    sequence = word_tokenize(sentence) 
    bigrams.extend(list(ngrams(sequence, 2)))

freq_dist = nltk.FreqDist(bigrams)
prob_dist = nltk.MLEProbDist(freq_dist)
number_of_bigrams = freq_dist.N()
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.