How to evaluate sequence to sequence models?

Question

I wonder how to evaluate variable long sequence-to-sequence predictions? Let us say I have the following $Y$ and $\hat{Y}$

$Y = [["1", "2", "2"], ["3", "2", "2"], ["1", "3", "2", "2"]]$

$\hat{Y} = [["1", "3", "2"], ["3", "3"], ["1", "3", "2"]]$

Shall I use a binary comparison where any mismatch counts as zero and any full match as one? Or shall calculate conventional accuracy by character-wise comparison?

My concern here is that on one hand, if this is a numeric predcition then any digit mismatch spoils the whole number so doesn't really matter where is the mismatch; on the other hand it would be nice which digits tend to miscalculated in order to find ways to improve the training sets.

Addition: the task is a numeric OCR so -- in contrast with a machine translation job where minor mistranslations are tolerable -- any digit mismatch could result significant business problem (different invoice sums for example). Moreover I'd like to know which individual digits tend to be misread more often so I need a way to get a statistic this way also.

Answer 1

Regarding your concern, there is no reason for you to choose only one evaluation metric. If there are several values that give you different views of the performance of the system, then compute all of these values. The evaluation should depend on your specific use case, so the important thing is that the values correlate with a good or bad performance of the system in the real world problem.

Even though I don't know exactly what your system is expected to do [written before question edit, see below for addition], maybe you could take as reference how performance is measured in speech and text recognition tasks. There you have a reference and a predicted sequence composed of characters and these characters form words. The length of the reference and the prediction is not necessarily the same. The performance is measured both at character and at word level (character error rate (CER) and word error rate (WER)). When measured at word level, even if only one character is wrong, the whole word is wrong. In both cases the main idea is to compute the levenshtein/edit distance (either at word or character level) between the reference and the prediction and then divide by the length of the reference.

In your example

Y=[["1","2","2"],["3","2","2"],["1","3","2","2"]]

Ŷ=[["1","3","2"],["3","3"],["1","3","2"]]

At inner bracket level the edit distances for the three inner brackets would be 1+2+1=4, since in the first a "3" is substituted by "2" (one edit) in the second a "3" is substituted by a "2" and a "2" is inserted (two edits) and in the last case again a "2" is inserted (one edit). The total length of the reference is 3+2+3=8. So the final performance measure is 4/8.

I am not sure if each of the inner brackets in your case are different samples (i.e., Ŷ is the whole dataset) or Ŷ is a single sample and you have something similar to characters and words. But anyway, if you do the edit distance between Ŷ and Y and divide by the length of Ŷ then you have 3/3, since all three brackets are different.

Additional response to question edit: If the task is number OCR, then you would compute character error rate. Computing an error value for each of the 10 digits is also possible with the edit distance. It would be by counting how many edit operations correspond to errors in each of the digits. For this in python you can use the editops function from the python-Levenshtein package (https://rawgit.com/ztane/python-Levenshtein/master/docs/Levenshtein.html#Levenshtein-editops).