1
$\begingroup$

I am stuck with a particular task of information extraction. I have a few hundred, long (5-35 pages) pdf, doc and docx project documents from which I seek to extract specific information and store them in a structured database.

The ultimate goal is to extract and store information in a way that we can query those and any new incoming documents for fast and reliable information. For instance, I want to query a combination of entities from the knowledge base and then return the n-most relevant paragraphs/sentences from the documents. Since some entities like “World Bank” are extracted dozens of times for some documents, I need a way to query the entity in context. Otherwise I just end up with a database that contains the names of specific entities without any way to map them back.

NER usually seems like a good solution for this, however, the documents all have very unique structures which also change from document to document. For instance, a lot of relevant information is stored in tables, but also in long paragraphs.

As far as I understand, NER uses the surrounding words to identify entities, hence loading in the whole documents as raw text and manually tagging terms in the tables will probably not serve as good training data.

For now I built a function that extracts the raw text from pdf, doc and docx documents and extracted entities with spaCys NER model, however, I need to define my own entities and the domain is to scientifc for spaCy to deliever good results. I also obtained a prodigy license for annotation.

Also, it seems I have to find a way to distinguish between "relevant" text and table entries and "junk" - aka footnotes, annexes, titles, subtitles, etc.

Any input is highly appreciated!

Thanks!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

If you think filtering certain POS-tag words or "bad" topics from topic modelling doesn't suffice to distinguish between relevant text and junk, you can implement your custom filter. Try regex rules and use your prodigy licence to annotate junk to fit a classifier based on tf-idf or other text feature extractor.

But don't forget, there isn't a preprocessing framework that fits all needs.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hi Adelson. Thank you for your input. Stemming and Lemmatization are usually part of NER models. For instance, building your own Named Entity Recognizer with spaCy or Prodigy automates all those steps. I think I was not clear with my questions, please excuse. I will edit it. $\endgroup$
    – XsLiar
    Commented Oct 15, 2019 at 13:19
  • $\begingroup$ ok, answer updated. $\endgroup$
    – Adelson Araújo
    Commented Oct 15, 2019 at 14:01
  • $\begingroup$ I thought about this, building a junk classifier in prodigy. However, I think it is hard to tell what is junk and what not. I opened a new question that goes into more detail about the last part of my problem: stackoverflow.com/questions/58416792/… $\endgroup$
    – XsLiar
    Commented Oct 17, 2019 at 13:24
1
$\begingroup$

One useful option is a vector database. The documents (or parts of the documents) can be stored as embeddings. Most vector databases support semantic search and approximate text matching.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.