What is the best approach for specified optical character recognition?

Question

I have a quite understandable request of extracting information (invoice number, invoice data, due date, total etc.) from scanned invoices (the digital format is image, not PDF), preferably in Python. The good thing is that the necessary information is more or less certain to exist on the page, and the (regexp-like) textual format of these is also tend to be consequent. The downside on the other hand is that the layout of the invoices are very diverse.

I have played with the following possible approaches:

• Use character recognition to extract pure text and later try to puzzle with the fragments. This method has some considerable problems: the quality of the OCR (at least of the implemented one in tesseract library) are quite mediocre and the output is hopelessly unstructured (practically a big pile of words), it is very difficult to come out any regexp or other rule even for regular phrases.
• My other approach would be to apply some kind of deep learning either to the raw image itself or the text pile where we leave the heavy lifting to the network, but in this case I'm not sure what is supposed to be the output? Is it some kind of a sequence to sequence mapping?

Very unusual task, indeed.

Answer 1

Optical character recognition is a well-studied problem with many possible solutions (ressources). CNNs have proven to work extremely well even for hand-written character recognition. Take a look at this two papers:

• Backpropagation Applied to Handwritten zip code
• Comparaison of Classifier Methods: A Case Study in Handwritten Digit Recognition

Here is a beginner tutorial to do just that with Tensorflow.

If you need extra data to train your model, take a look at the MNIST dataset.

Answer 2

One way could be applying a good segmentation technique and then classifying all the regions with some algorithms like(ann,svm,knn...), this method doesn't need a lot of training sample at all. Another way is applying deep learning (but this would mean a lot of training samples) you could use a CNN which perform a regression and so as output it gives you the text(this would mean you have to set a max sentences length). But to avoid a max sentences' length you could use a CNN with also some RNN layers and then you feed the network with parts of the image(a cascade approach).