Extracting information from table within invoices

Question

I am trying to find specific information from supplier invoices using ML. Now, I have a lot of invoices I can use for test (+100k). I figure the first step is to label them, so I can train a model to extract the information I need.

Below is a sample invoice (I have anonymized it by removing the shipper, logo etc.).

As you can see in the above image, I am trying to extract three entities from the invoice:

1. table
2. amount (for each line in the table)
3. tariff (for each line in the table)

Now I actually don't need the table, but what seems to be common across all my supplier invoices is that the amount and tariff for line imtes, is always within a table.

My idea of a pipeline is:

• Get all invoices (PDF files) and convert each page to a PNG image.
• Label all the individual pages, using a label annotation tool, to label table, amount and tariff - including the bbox information for each label.
• Train a model using the above dataset.
• ???

My questions are:

• Does this seem like a good approach to such a problem?
• Do I have to split up the model into two? With this I mean, does it make more sense to first identify the table within an image with one model, then serve the table image to another model, and extract the tariff and amount with the 2nd model?