Is NLP suitable for my legal contract parsing problem?

Question

My company has a product that involves the extraction of a variety of fields from legal contract PDFs. The current approach is very time consuming and messy, and I am exploring if NLP is a suitable alternative.

The PDFs that need to be parsed usually follow one of a number of "templates". Within a template, almost all of the documents are the same, except for 20 or so specific fields we are trying to extract. That being said, there are sometimes slight inconsistencies such as missing/added spaces, punctation, etc. Currently there are about 15 templates, but sometimes another gets added. Most of the data that needs to be extracted is in tabular form, but a few values are littered in paragraph text.

With the current, non-ML approach, we use a proprietary PDF parser that extracts out all of the tables into a json format. This parser must be custom configured for every new PDF template that appears, and this is time consuming. After converting to json, a custom regex expression must be made for each value in each template, leading to several hundred regexes total. Usually this approach is pretty accurate, but sometimes the slight inconsistencies can break it, meaning a the regex needs to be revised.

I'm wondering if NLP Named Entity Recognition would be a cleaner solution to this problem. My idea is to label all of the values in a few hundred sample documents and then train a custom NER model in a library like Spacy or Flair. Ideally, we could feed in the raw, extracted text from the PDFs instead of having to configure the custom parser to extract json.

The advantage I see in using the NLP approach is that we wouldn't have to configure the custom parser and write a bunch of regexes every time a new template appears. At worst, we would have to label a few of the new documents every time a template is added, which would presumably be faster and easier than the current approach. I think we could also generate tons of synthetic training data easily by swapping labeled values between different documents.

I am concerned that, using an ML approach, we wouldn't be able to achieve near perfect accuracy, which is a requirement. I'm also not sure how well NLP can perform on raw text from tables as opposed to paragraphs. The nice thing is that the documents tend to be very similar within templates.

I've never done NLP before, so I'm wondering if anyone here thinks this approach would be worthwhile exploring. If this is feasible, does anyone have suggestions on how to get the best results?

Answer 1

Here's what I would have done.

1. extract raw data from the the pdfs.
2. use rule based NER from spacy(it's quite fast), but it will require some manual pattern making. Also apply multiprocessing if you can. https://spacy.io/usage/rule-based-matching
3. You can also write a document classification to segregate the templates and send the templates to appropriate code,running parallely.
4. Also checkout blackstone library for legal texts, it might be of some use https://spacy.io/universe/project/blackstone

Answer 2

A couple of pictures of the documents with your target data highlighted might help. Also your approach seems sound enough. I agree you wouldn't get perfect results each time, however there is a concept of Human In the Loop. Where the data extracted by your program would have to be verified by a human once before certifying. This is basically a digital signature of the person verifying the data to have an audit trail. From what I understand, you're trying to extract just a few entities from the documents. So glancing over the data to verify whether things indeed make sense shouldn't be too difficult. This approach would certainly work faster than editing the template and making changes to the regular expressions.

Alternatively, your problem could also be construed as contextual summarization where the summary would be an extract of your NERs. I'm not sure how effective this would be (maybe someone more experienced could chime in) but you could go the deep learning route and train an attention based model like BERT on your custom dataset containing the extracted NERs as your target summarization.