Say you have a lot of PDF documents, say K documents.

Each document Di is Ni pages long. In one of the Ni pages (don't know which, say Pi), there is the information you need to extract.

I am thinking about building a three step pipeline for this:

Step 1:

Extract a random sample of Krand documents from your data.

Annotate all the pages into two categories (relevant/irrelevant)

Fine tune a document classification model on this data

Step 2:

Extract relevant pages from all documents using the document classification model.

Annotate a sample of relevant pages and fine tune an object detection model to recognize your Region of Interest in that document.

Step 3:

Run OCR on your ROIs.

Post-process the OCR outputs.

Will highly appreciate your comments and suggestions on which Document Classification model and Object Detection model will be the best and whether this is a right approach or not.


1 Answer 1


In my opinion, Document Classification could be a good approach, but I think it's vague. There are much better options now. One approach you can think of is that, create a vector database of all your documents, which will have all the data that you will need in the future. And then you can do topic modeling on it to get the information you need. The catch here is that, you will get vectors of all the text data, but to get vectors for the images, you need to get embeddings of those separately and append them with the database. One way to handle this is to use OCR on it and tune your OCR and post-process it add it with the text of the pdf.

Caution: This might be a good approach only if you need to extract more information than numbers only.

If you are only interested in handwritten numbers then I will suggest you to start with OCR first, tune the OCR as much as possible to fulfill your demands and then post process the outputs.

  • $\begingroup$ Yes, in fact I do. There is a lot of information I want to extract from the document. Are you saying I should run OCR and extract embedding from all the pages of the document? Every document has a lot of pages, and there are a lot of documents. @Harshad can I contact you to discuss this further? $\endgroup$
    – Anmol Deep
    Aug 29, 2023 at 5:16
  • $\begingroup$ Sure! Email me: [email protected] $\endgroup$ Aug 29, 2023 at 5:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.