How to extract handwritten phone numbers from a huge set of documents?

Question

Say you have a lot of PDF documents, say K documents.

Each document Di is Ni pages long. In one of the Ni pages (don't know which, say Pi), there is the information you need to extract.

I am thinking about building a three step pipeline for this:

Step 1:

Extract a random sample of Krand documents from your data.

Annotate all the pages into two categories (relevant/irrelevant)

Fine tune a document classification model on this data

Step 2:

Extract relevant pages from all documents using the document classification model.

Annotate a sample of relevant pages and fine tune an object detection model to recognize your Region of Interest in that document.

Step 3:

Run OCR on your ROIs.

Post-process the OCR outputs.

Will highly appreciate your comments and suggestions on which Document Classification model and Object Detection model will be the best and whether this is a right approach or not.

Answer 1

In my opinion, Document Classification could be a good approach, but I think it's vague. There are much better options now. One approach you can think of is that, create a vector database of all your documents, which will have all the data that you will need in the future. And then you can do topic modeling on it to get the information you need. The catch here is that, you will get vectors of all the text data, but to get vectors for the images, you need to get embeddings of those separately and append them with the database. One way to handle this is to use OCR on it and tune your OCR and post-process it add it with the text of the pdf.

Caution: This might be a good approach only if you need to extract more information than numbers only.

If you are only interested in handwritten numbers then I will suggest you to start with OCR first, tune the OCR as much as possible to fulfill your demands and then post process the outputs.