I'm trying to identify all the names in a novel (fed as a text file) using NLTK. On a smaller scale, the POS tagging works perfectly. However, when I feed a large body of text, by which I mean three or four paragraphs, the system fails miserably. How can I fix this?

I read the file, split it into lines, sentences and then words. What I get is a list of lists where each internal list contains words of a sentence. Then, I use NLTK to tag each sentence.

def process_file(_file, tagger, stemmer, stopwords, filename, printinfo):
    sentences = []
    _nnp = set()
    words = dict()
    for line in _file:
        for sentence in nltk.tokenize.sent_tokenize(line):
    sent_count = 0
    for sentence in sentences:
        tags = tagger.tag(sentence)
        for tag in tags:
            if tag[1] == "NNP" or tag[1] == "NNPS":
                if tag[0] not in stopwords:
                    stemmed_word = stemmer.stem(tag[0])
                    if stemmed_word not in words.keys():
                        words[stemmed_word] = 1
                        words[stemmed_word] += 1
        print("\r[{0}] Reading file '{1}'[{2:>3.1%}] ".format(printinfo, filename, sent_count/len(sentences)), end='')
return _nnp, words

Main code:

dictionary = dict()
nnp = set()

# Initialize tagger and stemmer
pos_tagger = nltk.tag.PerceptronTagger()
ps = nltk.PorterStemmer()
stopwords = nltk.corpus.stopwords.words('english')
file_count = 0

# For each file, extract words and named entities
for file in files:
    file_count += 1
    _nnp, _dictionary = process_file(open(file, 'r', encoding='utf-8'), pos_tagger, ps, stopwords, file, str(file_count)+"/"+str(len(files)))

    # Extend dictionary
    for word in _dictionary.keys():
        if word in dictionary.keys():
            dictionary[word] += _dictionary[word]
            dictionary[word] = _dictionary[word]

    # Join sets
    nnp = nnp.union(_nnp)
end = time.time()
print("COMPLETED: Step 2 completed in {0:.3f}s".format(end-start))

Here's a sample of what the text might look like:

In slow motion, afraid of what he was about to witness, Langdon rotated the fax 180 degrees. He looked at the word upside down.

Instantly, the breath went out of him. It was like he had been hit by a truck. Barely able to believe his eyes, he rotated the fax again, reading the brand right-side up and then upside down.

"Illuminati," he whispered.
  • $\begingroup$ What does fails miserably mean in this context? Are you getting an error message? Or simply unexpected output? $\endgroup$ Jul 3 '19 at 5:16
  • 1
    $\begingroup$ It classifies words like 'it' and other basic words as names. $\endgroup$ Jul 5 '19 at 10:52

I just processed a bunch of files from a random corpus and the results seem to make sense generally speaking, the entities list (at the bottom) does not contain any suspicious words -there may be an issue with false negatives, but there seems to be no obvious problem with false positives.

Any chance you mistook the keys of the variable dictionary for the contents of nnp? Check this out:

# print(dictionary.keys())
dict_keys(['thi', 'excel', 'start', 'film', 'career', 
'.', 'hi', 'talent', 'show', 'long', 'ahead', 'the', 'car',
 'truck', 'chase', 'excit', '1937', 'era', 'seri', 'american',
 'treasur', 'book', 'spring', 'perform', 'usual', 'owner', 
'right', ',', 'take', 'chanc', 'get', 'produc', 'I', 'think', 
'would', 'winner', 'recent', 'air', 'episod', "'s", 'It', 
# ... (skipping many items) ...
'bang-up', 'horror/act', 'hybrid'])

# print(nnp)
{'Better', 'Rackham', 'Unicorn', 'Rooney', 'Alucarda', 
'Mejia', 'Alejandra', 'Red', 'Soviets', 'Marisol', 'Remi', 
'Captain', 'Santacruz', 'Digest', 'Mickey', 'de', 'Infernal',
'PBS', 'Andy', 'Recently', 'England', 'Director/co-writer', 
'yummy', 'Anda', 'DVD', 'Mauri', "I'ts", 'Hergé.', 'Zero', 
'Alberto', 'Herge.', 'Baron', 'Byington', 'Sadoul', 'Galindo',
 'Mauricio', 'Tono', 'Free', 'Mexican', 'P.O.V', 'Boys', 
'Author', 'Swedish', 'Snowy', 'Breaking', 'Vietnam', 'TinTin',
 'Georges', 'Adriana', 'Antonio', 'Anyway', 'Remy', 'Charly', 
'Daniels', 'Mr', 'J.', 'Chan', 'A', 'Mr.', 'Plascencia', 
'Brilliantly', 'Congo', 'Pedro', '>', '<', 'Hardy', 'Hergé', 
'Awesome', 'Valentino', 'Blue', 'Please', 'Jesse', 'Gonzalez',
 'Fanny', 'Children', 'Tintin', 'Numa', 'Sea', 'Vega', 'Panic',
 '/', 'Herge', 'Edith', 'III', 'Catholic', 'Nacho', 'Lotus',
 'Sharks', 'CD', 'Fernandez'}


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.