# word/sentence alignment for English document

I have an English document, which is preprocessed into two versions. I want to align words or sentences from these two versions of the document. A simple example is as below:

I don't want to go there. My e-mail address ok.


should be aligned with

I do n't want to go there my email address ok


Or the tokens from the first sentence are aligned correctly with the tokens in the 2nd sentence. I have tried some sequence alignment methods, which do not perform well. Considering there are alignment tools for machine translation, automatic alignment for English should be easier right?

• Could you say what bio-sequence alignment methods you have tried? Sep 11 '18 at 2:45
• Actually not bio-sequence in particular, I have tried SequenceMatcher from difflib. I have also tried another one but had error running aligner. Do you have any experience on this? Sep 11 '18 at 9:52
• It looks like the most common mistakes are insertions of spaces and deletions of punctuation and replacing punctuation by spaces. So in particular s(" ","-") should be less of a penalty than s("a","p") in Smith-Waterman. That is the sort of parameter setting I wanted you to include in what you had already tried. Sep 11 '18 at 14:17

Use kaldi's align-text which align two sentences using Levenshtein distance.