So, this is how the problem goes: I am trying to extract information from scanned receipts like this,

Receipt from New York, USA

What I have been told is that I would get the textual data from a OCR software, so in short I will be working with a textual version of the image directly.


The problem in hand here is that I have to extract certain information here, namely,

  1. The Location. (for eg: New York, United States)
  2. The complete Total (after all the discounts, tips, etc. have been involved) (Eg: 1033.42)
  3. The Currency. ($, £, €, etc)
  4. The Date. (Easier to guess)

The reason I want to extract the Location information is that, if for example, the currency is not explicitly mentioned here, then I can infer from the location where the receipt is generated.


The challenge here is that, information like Total Due could be anything like Grand Total or Total (only), or something semantically similar to Total, as I am not going to get the same receipt from the same restaurant only. (The restaurant could be anywhere in the world, but the problem is limited to only English speaking countries for now.)

The other challenge is to actually get the total information. It's very easy for us to see that 1033.42 is the total in the above receipt. But how do I get the software to know that? The way I see it, is that 1033.42 is near to the total(proximity). But there could be other numbers near to it as well.

Where I have tried and failed:

I have been told to start from NLTK(NER) but NER doesn't work for everything here. I can get the date information through it but the problem isn't just only to identify what the named entities are, imo.

What I think would work

The way I see it, I think I need to use a machine/deep learning model where the machine would be able to understand the proximity match between anything which semantically says Total and the number near to it (most probably on the right side).

Any help regarding which model would work best in terms of speed(first and foremost) and accuracy would be greatly appreciated.
I would also appreciate any help regarding where I could find any dataset or existing model which I could use for Transfer Learning.

  • $\begingroup$ Are you able to do it? $\endgroup$
    – Pygirl
    Dec 29, 2020 at 7:29
  • $\begingroup$ I did it in a different way. Used regex for Total and Date and Libpostal library for addresses and Currency. $\endgroup$
    – Zaid Khan
    Dec 29, 2020 at 7:35
  • $\begingroup$ Did you try out some of the deep learning technique? How was the performance using regex? $\endgroup$
    – Pygirl
    Dec 29, 2020 at 7:40
  • $\begingroup$ Deep learning? I think the library I used Libpostal and Google Vision API maybe Deep Learning libraries. Idk. The performace was pretty average. I did not expect better though because the receipts were quite heterogeneous in nature $\endgroup$
    – Zaid Khan
    Dec 29, 2020 at 7:46
  • $\begingroup$ Nope I mean to say using PICK, bertgrid etc $\endgroup$
    – Pygirl
    Dec 29, 2020 at 7:51

1 Answer 1


There is already an ML engine that does these extractions, here is a general process layout: There

Hre is the orginal paper describing the architecture, features, approaches etc. read it up, instead of me copying it here. cloudscan

  • $\begingroup$ Hi. Thank you for the answer. I had a long look at the paper. I understood a lot of things but the problem I'm finding is that I don't have a large dataset which the authors have trained the model on (Some 3 hundred thousand receipts). I have searched a lot on the Internet but I haven't found it anywhere. Could you help here? $\endgroup$
    – Zaid Khan
    Jan 15, 2020 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.