I am dealing with the task to extract structured information from domain-specific unstructured documents. The end goal is to obtain a reliable, queryable system, i.e. in the form of a chat-bot or Question-Answering application.

During my research I formulated following solution approach:

  1. Parse doc, docx and pdf documents to raw text. Pre-process with common NLP tools.

  2. Consult with client and decide upon several categories for entities.

  3. Train customised NER model to obtain entities from texts.

  4. Obtain entity-level-relations.

  5. Populate knowledge base

  6. Migrate knowledge base data to knowledge graph and set up the queryable system.

For the implementation, I was looking into Stanfords Deepdive architecture, however they stopped support in 2017 and I am not sure if it is still up to date.

Alternatively, I intended to use spaCy to train my own NER model, but it is unclear to me if this allows to obtain the entity-level-relations for populating a knowledge base.

Could you advise me to alternatives to Deepdive or any other resources that demonstrate how to build up such a system from scratch?

  • $\begingroup$ relation extraction is the most difficult part here, I'm not sure there's any standard, domain-independent approach for that, let alone good software. For what it's worth I would still have a serious look at DeepDive, that would still give you a good basis for tuning your specific system. $\endgroup$
    – Erwan
    Nov 22, 2019 at 1:29


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.