One way is to loop through a list of sentences. Process each one
sentence separately and collect the results
Brian is totally right about the solution but I think that's actually the part that is missing from his answer :) Brian's code assumes that the sentences have already been segmented, which was Ahmad's original question as far as I can tell.
With that in mind, here's an updated proposal for adding the sentence splitting to the script:
import itertools, nltk
def ngrams_wrapper(sent):
return list(nltk.ngrams(sent, 2))
raw = "To Sherlock Holmes she is always the woman." + \
"I have seldom heard him mention her under any other name."
# Given a string as input (`raw`), split it into sentences.
# Returns an object of type 'list: str'
sentences = nltk.sent_tokenize(raw)
# ['To Sherlock Holmes she is always the woman.',
# 'I have seldom heard him mention her under any other name.']
# Next, tokenize every sentence (string) in the list of sentences. The tokenizer takes
# strings as input so we need to apply it on each element of `sentences` (we can't apply
# it on the list itself). For that, we can use the function `map`, which applies any
# callable Python object to every element of a list.
# The output of this step will be an object of type
# 'list: list: str'
# such that
# 'sentences: sentence: word'
tokenized = map(nltk.tokenize.word_tokenize, sentences)
# [['To', 'Sherlock', 'Holmes', 'she', 'is', 'always', 'the', 'woman', '.'],
# ['I', 'have', 'seldom', 'heard', 'him', 'mention', 'her', 'under', 'any', 'other',
# 'name', '.']]
# We now generate the ngrams for each tokenized sentence in the list of sentences. Since
# we want to generate the ngrams for each sentence specifically, we again must apply the
# function to each element of the list separately, not the whole list. So, we use the
# `map` function again.
# The output of this step will be an object of type:
# 'list: list: tuple: <str, str>'
# such that
# 'sentences: sentence: ngram: <word_1, word_2>
bigrams = map(ngrams_wrapper, tokenized)
# [[('To', 'Sherlock'), ('Sherlock', 'Holmes'), ('Holmes', 'she'), ('she', 'is'), ('is',
# 'always'), ('always', 'the'), ('the', 'woman'), ('woman', '.')], [('I', 'have'),
# ('have', 'seldom'), ('seldom', 'heard'), ('heard', 'him'), ('him', 'mention'),
# ('mention', 'her'), ('her', 'under'), ('under', 'any'), ('any', 'other'), ('other',
# 'name'), ('name', '.')]]
# Finally, since we want all the bigrams in a single list to work with them more easily,
# we flatten the list, that is, we concatenate all the elements at the first level of
# depth in the list. So, if the list was an object of type
# 'list: list: tuple: <str, str>' (sentences: sentence: ngram)
# we will transform it into:
# 'list: tuple: <str, str>' (sentences: ngram)
bigram = list(itertools.chain.from_iterable(bigrams))
# [('To', 'Sherlock'), ('Sherlock', 'Holmes'), ('Holmes', 'she'), ('she', 'is'), ('is',
# 'always'), ('always', 'the'), ('the', 'woman'), ('woman', '.'), ('I', 'have'), ('have',
# 'seldom'), ('seldom', 'heard'), ('heard', 'him'), ('him', 'mention'), ('mention',
# 'her'), ('her', 'under'), ('under', 'any'), ('any', 'other'), ('other', 'name'),
# ('name', '.')]
freq_dist = nltk.FreqDist(bigram)
# [(('always', 'the'), 1), (('woman', '.'), 1), (('Holmes', 'she'), 1), (('seldom',
# 'heard'), 1), (('Sherlock', 'Holmes'), 1), (('him', 'mention'), 1), (('I', 'have'), 1),
# (('any', 'other'), 1), (('under', 'any'), 1), (('the', 'woman'), 1), (('her', 'under'),
# 1), (('other', 'name'), 1), (('To', 'Sherlock'), 1), (('name', '.'), 1), (('she', 'is'),
# 1), (('heard', 'him'), 1), (('mention', 'her'), 1), (('have', 'seldom'), 1), (('is',
# 'always'), 1)]
prob_dist = nltk.MLEProbDist(freq_dist)
number_of_bigrams = freq_dist.N()