Python: validating the existence of NLTK data with database search

Question

I need to pull the names of companies out of resumes. Thousands of them. I was thinking of using NLTK to create a list of possible companies, and then cross-referencing the list of strings with something like SEC.gov.

I've already been able to successfully pull the candidate's name, and contact info off of the resumes with some RegEx, but this one has me quite stumped.

What I'm thinking is that I could use NLTK to create a list of strings of proper nouns from the resume's, and then search SEC.gov, or some other database.

This is a link to the SEC page I would be searching: SEC company search page

Read Resume1
Get all potential company names as list of strings potentialCompanies
    IF searching for string1 in SEC gets result, THEN add to candidateCompanies
        ELSE remove from potentialCompanies, go to next string

My Questions

To people that have used NLTK, would there be a better way of getting the potential companies from the text besides using proper nouns?

Would there be a better place to search for companies than the SEC site?

I have never done any web scraping before, and don't really know where to start if it is needed.

(I had posted this on Stack Overflow but they told me that it might be better suited for here...)

Answer 1

NLTK has a built-in NER model that would extract potential Organizations from text, you can read about it here (and see examples) NLTK book (look for section "5 Named Entity Recognition").

However, if your input text has organizations in a very specific context that wasn't seen by NLTK NER model, performance might be quite low. In that case you should be looking into training your own NER model, what would extract company names. For that you would require to manually markup a small amount of your dataset.