# How to create a model to recognize matching label and ROI with OCR

I am trying to make a model in python using Tensorflow to process the Tesseract OCR to detecting and extracting particular ROI from Image. I want to recognize the particular fields and values from Invoice with our model. For example i want to extract the Totoal amount and itemized amount and prices which is in tabular format. I want to grab only those details with our model. I am able to grab the 4 ROI cordinates of label with its's value manual in OCR. I am able to achieve these so far:

• Four coordinates with OCR output for specific ROI in image
• My sample dataset is like in this format [231, 404, 352, 616, 'Total Amount'] where first four values are coordinates and the last one is a label.
• I am doing manually with mouse selection with my custom made python script where i select the label and all its value in particular area, row/columns with values.

This is a sample of image:

Now i want to train and prepare a model so that when i test with other invoice image having different value my model can detect and extract only that particular area on image with label and all those values either in row or column.

Now my question is How can i achieve this, How to train a model to detect only those area having label with Total Amount and all its's value present in that area on invoice image. Thanks.

• What do you mean by ROI ? For me it means Return on investment... Feb 18, 2020 at 9:06
• No. It is Region of Interest. I mean the desired area in particular image to be extracted with Tesseract OCR. Feb 18, 2020 at 9:21

Get the pixels of that word - run image_to_data method of ocr to get pixels.