I'm trying to find metrics to evaluate multiple algorithms for key information extraction from already OCRed invoices. For instance, such an algorithm, given an invoice, could find that:

    "company": "STARBUCKS STORE #10208",
    "date": "14/03/2015",
    "address": "11302 EUCLID AVENUE, CLEVELAND, OH (216) 229-0749",
    "total": "4.95",

Formally, I defined the task as a function f that for a given invoice document D returns a dictionary of entries: {field1:value1,field2:value2,...}.

Initially, I thought about using precision, recall and F1 measures, assuming that each dictionary entry is marked as correct if both a field and its value matches exactly the ground truth. I borrowed this approach from ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction.

However, the problem with metrics based on exact string match is that they penalize even small errors: missing/redundant words, single differences in characters and spacing. Those errors may stem not only from the imperfect information extraction algorithm but also from faulty OCR algorithm. Unfortunately, I am not able to test information extraction in isolation (i.e., given 100% perfectly OCRed input).

Therefore, I am looking for some more relaxed metrics. What are my possibilities? What has been used in this field?

Some ideas that come to my mind:

  • calculate distance between two texts using Levenstein distance on character/word
  • consider two texts matching if distance is smaller than threshold X.
  • or include information about the distance in the final measures, e.g. normalize it and micro-average over each document.
  • 1
    $\begingroup$ For a project I used your last supposed approach "or include information about the distance in the final measures, e.g. normalize it and micro-average over each document." - e.g. delete all symbols, use sensitive letter case + strip individual words, etc. and then used levenshtein for each entry. I accomplished some solid results. $\endgroup$
    – Maeaex1
    Dec 4 '19 at 10:32
  • $\begingroup$ @Maeaex1 Sounds good. What was your ultimate metric between a predicted and expected (ground through) document. Similarity? Average over similarities of fields? And than effectiveness of the algorithm would be a similarity averaged over similarities of verified documents. $\endgroup$
    – dzieciou
    Dec 4 '19 at 11:09

I have worked with Structured text using OCR. OCR is prone to errors even while reading the content and a slight change in string arrangement would lead to false positives. I used Cosine Similarity and LEvenstein distance. I curated the data with pip install ftfy, Flashtext fuzzywuzzy ,chardet and set a threshold of 97% giving with 2% buffer for errors which might be unintentional/Typos or Inability of OCR to read the character etc. With this approach you are just relaxing on the 2% error and you can still use the same Precision, Recall or Specificity and Accuracy metrics. And we based on new data, you can gather observations and adjust your threshold on run time. And yes, Make your threshold configurable than hard coded which will be easier to run multiple experiments.

I hope this helps.

  • $\begingroup$ By cosine similarity do you mean cosine similarity between TF/IDF vectors of fields values? I wonder if for such short field values (up to 10 words at most) that makes sense in my case. $\endgroup$
    – dzieciou
    Dec 4 '19 at 11:00
  • 1
    $\begingroup$ Yes, You can. In the end, its mathematics thats doing the magic in the background. You can experiment with N-Grams,BOW, etc You can refer to the below links and you can also calculate on different N-Grams to find contextual information. A single token might not give you the real context but with N-grams, you can capture the preceding and succeeding word. stackoverflow.com/questions/55067412/… $\endgroup$
    – Syenix
    Dec 4 '19 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.