# A few ideas to parse "events" from a text document

I'm working with lots of press releases (mostly PDF or DOC documents, i.e. text). I would like to automatically parse them into a CSV file, 1 line per event.

Input:

Output:

datetime;location;title;description
2016-06-02 18:30;Public Library;About Picasso's last years;Lorem ipsum...
2016-06-03 17:30;Public Library;1001 stories;For children, 3 to 6 year old


I could write a parser for each precise layout of document (e.g. date always in bold, description always in italic) or detect keywords (example: january, february, etc. => assume the nearby words are a date, etc.). But this would require lots of work for every different press release.

I'm looking for more clever solutions, maybe involving machine learning and/or neural networks. I'm open to take a MOOC on this subject (I have a math PhD background and programming skills).

• What general techniques do you see to address such a big (and difficult topic) concern ?

• Are there open-source projects dedicated to that?

• Some Python libraries that would make this task easier?

• Any general idea?

These documents can be very unstructured and machine learning rather takes perfectly structured data like a single clean table of numbers. The real hard part, how to translate all the interesting information (font size, bold, position, order, ...) into a clean table, isn't solved by machine learning. You see it's hard to create a sensible, purely numeric table from a document.

I think your best bet is to use a natural language processing toolkit and do named entity recognition. See http://www.nltk.org/book/ch07.html "5 Named entity recognition". Or see the recommendation here: https://stackoverflow.com/questions/11333903/nltk-named-entity-recognition-with-custom-data

With this auxiliary information, I'd rather spend time handcoding rules, which work on most documents. You can create a nested data structure and try to match patterns like "headline, enumitem1, enumitem2, ..."

The rules would throw errors on documents where they are unsure. For these documents you will have to manually check what's going on. But this way you don't lose information.

I'm no expert on the very advanced methods, but in theory, with a lot of labeled data, a very clever encoding schema and a fair bit of luck an approach like RNN might be successful. But even if all the mentioned requirements hold, the training of those RNNs require a lot of experience that is hard to learn from references.

• Thanks. Two small questions : what is RNN? do you know some open-source project that could do such crawling, that it would be worth trying before diving into complex algorithms?
– Basj
Jun 5, 2016 at 19:26
• Recurrent Neural Network. But it probably takes one experienced fulltime PhD for a couple of months to make it work - if at all possible. I'd recommend the two above mentioned libraries to help detecting interesting entities. Apart from that, PDF or DOC reading libraries, of course (havent tried any). And in general, definitely using Python with all it's data structures and maybe regex. No sure what would be worth trying beyond that. I doubt anyone has a general solution for your problem yet, but it can be possible to some extend. Jun 5, 2016 at 21:53
• I can validate what @Gerenuk said. I know people who are paid big bucks to handle unstructured text. It is not an easy problem. I would use a natural language toolkit with a lot of hard-coded heuristics thrown in. Jun 9, 2016 at 22:57

Try Conditional random fields. CRFs are a class of statistical modeling method often applied in pattern recognition and machine learning and used for structured prediction. There are several open source implemenation available CRF++, CRFSuit. The core idea is using context as feature for solving sequence labeling problem.

RNN (LSTM) can be another apporach. But its a kind of black box and you don't have much control over the features to learn.