I have a quite understandable request of extracting information (invoice number, invoice data, due date, total etc.) from scanned invoices (the digital format is image, not PDF), preferably in Python. The good thing is that the necessary information is more or less certain to exist on the page, and the (regexp-like) textual format of these is also tend to be consequent. The downside on the other hand is that the layout of the invoices are very diverse.

I have played with the following possible approaches:

  • Use character recognition to extract pure text and later try to puzzle with the fragments. This method has some considerable problems: the quality of the OCR (at least of the implemented one in tesseract library) are quite mediocre and the output is hopelessly unstructured (practically a big pile of words), it is very difficult to come out any regexp or other rule even for regular phrases.
  • My other approach would be to apply some kind of deep learning either to the raw image itself or the text pile where we leave the heavy lifting to the network, but in this case I'm not sure what is supposed to be the output? Is it some kind of a sequence to sequence mapping?

Very unusual task, indeed.

  • $\begingroup$ Search terms: "Document layout analysis" followed by "document image information extraction". $\endgroup$
    – Emre
    Commented Mar 23, 2017 at 8:56
  • $\begingroup$ @Hendrik What do you mean by the layout is very 'diverse'? I am assuming you mean that their is no standard layout format. If you gave examples, I bet you would get even better responses $\endgroup$ Commented Sep 17, 2017 at 8:11

2 Answers 2


Optical character recognition is a well-studied problem with many possible solutions (ressources). CNNs have proven to work extremely well even for hand-written character recognition. Take a look at this two papers:

Here is a beginner tutorial to do just that with Tensorflow.

If you need extra data to train your model, take a look at the MNIST dataset.

  • $\begingroup$ How does this answer his question? $\endgroup$ Commented Sep 17, 2017 at 8:12

One way could be applying a good segmentation technique and then classifying all the regions with some algorithms like(ann,svm,knn...), this method doesn't need a lot of training sample at all. Another way is applying deep learning (but this would mean a lot of training samples) you could use a CNN which perform a regression and so as output it gives you the text(this would mean you have to set a max sentences length). But to avoid a max sentences' length you could use a CNN with also some RNN layers and then you feed the network with parts of the image(a cascade approach).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.