# Extracting name, date and total from a set of heterogeneous receipts

So, this is how the problem goes: I am trying to extract information from scanned receipts like this,

What I have been told is that I would get the textual data from a OCR software, so in short I will be working with a textual version of the image directly.

Problem:

The problem in hand here is that I have to extract certain information here, namely,

1. The Location. (for eg: New York, United States)
2. The complete Total (after all the discounts, tips, etc. have been involved) (Eg: 1033.42)
3. The Currency. (\$, £, €, etc)
4. The Date. (Easier to guess)

The reason I want to extract the Location information is that, if for example, the currency is not explicitly mentioned here, then I can infer from the location where the receipt is generated.

Challenges:

The challenge here is that, information like Total Due could be anything like Grand Total or Total (only), or something semantically similar to Total, as I am not going to get the same receipt from the same restaurant only. (The restaurant could be anywhere in the world, but the problem is limited to only English speaking countries for now.)

The other challenge is to actually get the total information. It's very easy for us to see that 1033.42 is the total in the above receipt. But how do I get the software to know that? The way I see it, is that 1033.42 is near to the total(proximity). But there could be other numbers near to it as well.

Where I have tried and failed:

I have been told to start from NLTK(NER) but NER doesn't work for everything here. I can get the date information through it but the problem isn't just only to identify what the named entities are, imo.

What I think would work

The way I see it, I think I need to use a machine/deep learning model where the machine would be able to understand the proximity match between anything which semantically says Total and the number near to it (most probably on the right side).

Any help regarding which model would work best in terms of speed(first and foremost) and accuracy would be greatly appreciated.
I would also appreciate any help regarding where I could find any dataset or existing model which I could use for Transfer Learning.

• Are you able to do it? Commented Dec 29, 2020 at 7:29
• I did it in a different way. Used regex for Total and Date and Libpostal library for addresses and Currency. Commented Dec 29, 2020 at 7:35
• Did you try out some of the deep learning technique? How was the performance using regex? Commented Dec 29, 2020 at 7:40
• Deep learning? I think the library I used Libpostal and Google Vision API maybe Deep Learning libraries. Idk. The performace was pretty average. I did not expect better though because the receipts were quite heterogeneous in nature Commented Dec 29, 2020 at 7:46
• Nope I mean to say using PICK, bertgrid etc Commented Dec 29, 2020 at 7:51