# Evaluating information extraction from structured documents

I'm trying to find metrics to evaluate multiple algorithms for key information extraction from already OCRed invoices. For instance, such an algorithm, given an invoice, could find that:

{
"company": "STARBUCKS STORE #10208",
"date": "14/03/2015",
"address": "11302 EUCLID AVENUE, CLEVELAND, OH (216) 229-0749",
"total": "4.95",
}


Formally, I defined the task as a function f that for a given invoice document D returns a dictionary of entries: {field1:value1,field2:value2,...}.

Initially, I thought about using precision, recall and F1 measures, assuming that each dictionary entry is marked as correct if both a field and its value matches exactly the ground truth. I borrowed this approach from ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction.

However, the problem with metrics based on exact string match is that they penalize even small errors: missing/redundant words, single differences in characters and spacing. Those errors may stem not only from the imperfect information extraction algorithm but also from faulty OCR algorithm. Unfortunately, I am not able to test information extraction in isolation (i.e., given 100% perfectly OCRed input).

Therefore, I am looking for some more relaxed metrics. What are my possibilities? What has been used in this field?

Some ideas that come to my mind:

• calculate distance between two texts using Levenstein distance on character/word
• consider two texts matching if distance is smaller than threshold X.
• or include information about the distance in the final measures, e.g. normalize it and micro-average over each document.
• For a project I used your last supposed approach "or include information about the distance in the final measures, e.g. normalize it and micro-average over each document." - e.g. delete all symbols, use sensitive letter case + strip individual words, etc. and then used levenshtein for each entry. I accomplished some solid results. – Maeaex1 Dec 4 '19 at 10:32
• @Maeaex1 Sounds good. What was your ultimate metric between a predicted and expected (ground through) document. Similarity? Average over similarities of fields? And than effectiveness of the algorithm would be a similarity averaged over similarities of verified documents. – dzieciou Dec 4 '19 at 11:09