A few ideas to parse "events" from a text document

Question

I'm working with lots of press releases (mostly PDF or DOC documents, i.e. text). I would like to automatically parse them into a CSV file, 1 line per event.

Input:

Output:

datetime;location;title;description
2016-06-02 18:30;Public Library;About Picasso's last years;Lorem ipsum...
2016-06-03 17:30;Public Library;1001 stories;For children, 3 to 6 year old

---

I could write a parser for each precise layout of document (e.g. date always in bold, description always in italic) or detect keywords (example: january, february, etc. => assume the nearby words are a date, etc.). But this would require lots of work for every different press release.

I'm looking for more clever solutions, maybe involving machine learning and/or neural networks. I'm open to take a MOOC on this subject (I have a math PhD background and programming skills).

• What general techniques do you see to address such a big (and difficult topic) concern ?

• Are there open-source projects dedicated to that?

• Some Python libraries that would make this task easier?

• Any general idea?

Answer 1

These documents can be very unstructured and machine learning rather takes perfectly structured data like a single clean table of numbers. The real hard part, how to translate all the interesting information (font size, bold, position, order, ...) into a clean table, isn't solved by machine learning. You see it's hard to create a sensible, purely numeric table from a document.

I think your best bet is to use a natural language processing toolkit and do named entity recognition. See http://www.nltk.org/book/ch07.html "5 Named entity recognition". Or see the recommendation here: https://stackoverflow.com/questions/11333903/nltk-named-entity-recognition-with-custom-data

With this auxiliary information, I'd rather spend time handcoding rules, which work on most documents. You can create a nested data structure and try to match patterns like "headline, enumitem1, enumitem2, ..."

The rules would throw errors on documents where they are unsure. For these documents you will have to manually check what's going on. But this way you don't lose information.

I'm no expert on the very advanced methods, but in theory, with a lot of labeled data, a very clever encoding schema and a fair bit of luck an approach like RNN might be successful. But even if all the mentioned requirements hold, the training of those RNNs require a lot of experience that is hard to learn from references.

Answer 2

Try Conditional random fields. CRFs are a class of statistical modeling method often applied in pattern recognition and machine learning and used for structured prediction. There are several open source implemenation available CRF++, CRFSuit. The core idea is using context as feature for solving sequence labeling problem.

RNN (LSTM) can be another apporach. But its a kind of black box and you don't have much control over the features to learn.