Hello good NLP people,

I am working on a task that gradually seems not solvable for me. My data-set consists of long, messy, unstructured documents (pdfs, doc, docx, scans with tables, graphs, text, etc) and the client wants to obtain a system that allows to query basically as much information as possible from the documents.

An example query could be: “Who are the beneficiaries of project xxx that is implemented by ORG xxx?”. Or: “How much co-financing was directed to projects that concern focal areas x,y,z?”

My initial idea was following:

1) Process documents (Python, OCR, etc) into machine readable form.

2) Pre-process/clean/normalise text.

3) Manually annotate entities and build customised NER model.

4) Manually annotate entity relations and build Relation extraction model (I am not sure how to do this the best way?)

5) Extract triples.

6) Store triples in knowledge graph/base.

7) Query (however the query will then be limited by the extracted entities and relations).

The end result could look like this: http://semanticparsing.ukp.informatik.tu-darmstadt.de:5000/question-answering/static/index.html

However, I have the feeling that QA systems and Relation Extraction frameworks can be built on top of large quality datasets (like Wikipedia - or even better KB like Wikidata). In reality, dealing for example with messy documents, it seems very difficult to replicate.

Please let me know about any important work on this lately, or how you would proceed. Thanks!

  • $\begingroup$ This is doable, but is certainly non trivial. To be honest, I would recommend looking at projects like MarkLogic and Elasticsearch. I'm not sure how well they could handle "scans with tables" and the other data sources but its a start at least? $\endgroup$
    – Dylan
    Jan 10 '20 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.