# How can you build a model that extracts data out from receipts?

I'm trying to build a model that is capable of identifying information on receipts and invoices.

I have used google cloud vision api for text extraction from the receipt but the problem is it just returns all the text from a receipt. I am looking to build a model that returns only a certain fields such as total price, brought alcohol etc.. from a receipt.

I could parse the text to extract by hard coding things but it's not optimal I think. Is there any way to build something for this use case. I need just something to go on.

Comment --- Output:

'Berghotel\nGrosse Scheidegg\n3818 Grindelwald\nFamilie R. Müller\nRech. Nr. 4572\nBar\n30.07.2007/13:29:17\nTisch 7/01\nNM\n#ರ\n2xLatte Macchiato à 4.50 CHF\n1xGloki\nà 5.00 CHF\n1xSchweinschnitzel à 22.00 CHF\n1xChässpätzli à 18.50 CHF\n#ರ #ರ #1ರ\n5.00\n22.00\n18.50\nTotal:\nCHF\n54.50\nIncl. 7.6% MwSt\n54.50 CHF:\n3.85\nEntspricht in Euro 36.33 EUR\nEs bediente Sie: Ursula\nMwSt Nr. : 430 234\nTel.: 033 853 67 16\nFax.: 033 853 67 19\nE-mail: grossescheidegg@bluewin.ch\n'


The simplest pipeline would be to do the following:

1. OCR
2. Named Entity Extraction
3. Entity Disambiguation

# OCR

This is basically transforming your receipts into plain text.

If you have scans (pictures) of the receipts, then you need a method that can deal with images. For example, you could use tesseract.

# Named Entity Extraction

This is detecting the parts of interest in the text. This includes detecting dates, prices, currency, locations, names, and so on. Once you detect a named entity, for example: price=2.59, you still don't know if that is the unit cost, or total cost after tax, or total cost before tax, but that will come in the next step. All you know that this is an area of interest.

There are tools that are able to detect most named entities that you might want to use. For example: spaCy. The cool thing about spaCy is that you can train it to recognize more named entities that you might want.

# Entity disambiguation

This is identifying the purpose of the named entity that you extracted. For example, given price=2.59, is it the unit cost, or total cost after tax, or total cost before tax.

This is where rules might come in handy, because it will be difficult to teach a machine learning model to identify them. You can have rules like:

1. highest value = total cost after tax
2. unity quantity × unit price = total cost before tax
3. total cost before tax × 1.21 = total cost after tax
4. and so on...
• thank you so much for helping me out. I'm stuck with this implementation for weeks now. Can you explain the entity extraction a little more in depth so I can actually implement it, actually I have receipts like this and using ocr api's I was able to extract all the raw text from the receipt (check my question I've posted output) but the problem is I only need to extract few things like total price, shop name..... Here I don't have price=2.59 so and so and I don't want to hard code things like using regex. What to do? – user_12 Dec 28 '19 at 18:02
• @user_12 yes, you don't need to hardcode regex for extracting. The price=2.59 is just an example. You can use this link to see how to use spaCy for entity extraction. Here is a list of the pre-trained entities it knows. – BrunoGL Dec 28 '19 at 18:46
• It's just that I've never implemented any entity extraction model. Have you seen the output that I've got after text extraction. Is it possible to extract price, product name and price from such texts that I've received? – user_12 Dec 28 '19 at 19:14