I'm working with lots of press releases (mostly PDF or DOC documents, i.e. text). I would like to automatically parse them into a CSV file, 1 line per event.

Input: enter image description here


2016-06-02 18:30;Public Library;About Picasso's last years;Lorem ipsum...
2016-06-03 17:30;Public Library;1001 stories;For children, 3 to 6 year old

I could write a parser for each precise layout of document (e.g. date always in bold, description always in italic) or detect keywords (example: january, february, etc. => assume the nearby words are a date, etc.). But this would require lots of work for every different press release.

I'm looking for more clever solutions, maybe involving machine learning and/or neural networks. I'm open to take a MOOC on this subject (I have a math PhD background and programming skills).

  • What general techniques do you see to address such a big (and difficult topic) concern ?

  • Are there open-source projects dedicated to that?

  • Some Python libraries that would make this task easier?

  • Any general idea?


These documents can be very unstructured and machine learning rather takes perfectly structured data like a single clean table of numbers. The real hard part, how to translate all the interesting information (font size, bold, position, order, ...) into a clean table, isn't solved by machine learning. You see it's hard to create a sensible, purely numeric table from a document.

I think your best bet is to use a natural language processing toolkit and do named entity recognition. See http://www.nltk.org/book/ch07.html "5 Named entity recognition". Or see the recommendation here: https://stackoverflow.com/questions/11333903/nltk-named-entity-recognition-with-custom-data

With this auxiliary information, I'd rather spend time handcoding rules, which work on most documents. You can create a nested data structure and try to match patterns like "headline, enumitem1, enumitem2, ..."

The rules would throw errors on documents where they are unsure. For these documents you will have to manually check what's going on. But this way you don't lose information.

I'm no expert on the very advanced methods, but in theory, with a lot of labeled data, a very clever encoding schema and a fair bit of luck an approach like RNN might be successful. But even if all the mentioned requirements hold, the training of those RNNs require a lot of experience that is hard to learn from references.

  • $\begingroup$ Thanks. Two small questions : what is RNN? do you know some open-source project that could do such crawling, that it would be worth trying before diving into complex algorithms? $\endgroup$ – Basj Jun 5 '16 at 19:26
  • $\begingroup$ Recurrent Neural Network. But it probably takes one experienced fulltime PhD for a couple of months to make it work - if at all possible. I'd recommend the two above mentioned libraries to help detecting interesting entities. Apart from that, PDF or DOC reading libraries, of course (havent tried any). And in general, definitely using Python with all it's data structures and maybe regex. No sure what would be worth trying beyond that. I doubt anyone has a general solution for your problem yet, but it can be possible to some extend. $\endgroup$ – Gerenuk Jun 5 '16 at 21:53
  • $\begingroup$ I can validate what @Gerenuk said. I know people who are paid big bucks to handle unstructured text. It is not an easy problem. I would use a natural language toolkit with a lot of hard-coded heuristics thrown in. $\endgroup$ – Ricardo Cruz Jun 9 '16 at 22:57

Try Conditional random fields. CRFs are a class of statistical modeling method often applied in pattern recognition and machine learning and used for structured prediction. There are several open source implemenation available CRF++, CRFSuit. The core idea is using context as feature for solving sequence labeling problem.

RNN (LSTM) can be another apporach. But its a kind of black box and you don't have much control over the features to learn.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.