# How to evaluate sequence to sequence models?

I wonder how to evaluate variable long sequence-to-sequence predictions? Let us say I have the following $Y$ and $\hat{Y}$

$Y = [["1", "2", "2"], ["3", "2", "2"], ["1", "3", "2", "2"]]$

$\hat{Y} = [["1", "3", "2"], ["3", "3"], ["1", "3", "2"]]$

Shall I use a binary comparison where any mismatch counts as zero and any full match as one? Or shall calculate conventional accuracy by character-wise comparison?

My concern here is that on one hand, if this is a numeric predcition then any digit mismatch spoils the whole number so doesn't really matter where is the mismatch; on the other hand it would be nice which digits tend to miscalculated in order to find ways to improve the training sets.

Addition: the task is a numeric OCR so -- in contrast with a machine translation job where minor mistranslations are tolerable -- any digit mismatch could result significant business problem (different invoice sums for example). Moreover I'd like to know which individual digits tend to be misread more often so I need a way to get a statistic this way also.

• can you explain a bit more on the business case? because I think Association Mining can be used over here. can give you clear explanation if it suits your business problem. Dec 11, 2017 at 10:07

Regarding your concern, there is no reason for you to choose only one evaluation metric. If there are several values that give you different views of the performance of the system, then compute all of these values. The evaluation should depend on your specific use case, so the important thing is that the values correlate with a good or bad performance of the system in the real world problem.

Even though I don't know exactly what your system is expected to do [written before question edit, see below for addition], maybe you could take as reference how performance is measured in speech and text recognition tasks. There you have a reference and a predicted sequence composed of characters and these characters form words. The length of the reference and the prediction is not necessarily the same. The performance is measured both at character and at word level (character error rate (CER) and word error rate (WER)). When measured at word level, even if only one character is wrong, the whole word is wrong. In both cases the main idea is to compute the levenshtein/edit distance (either at word or character level) between the reference and the prediction and then divide by the length of the reference.