I have an English document, which is preprocessed into two versions. I want to align words or sentences from these two versions of the document. A simple example is as below:

I don't want to go there. My e-mail address ok.

should be aligned with

I do n't want to go there my email address ok

Or the tokens from the first sentence are aligned correctly with the tokens in the 2nd sentence. I have tried some sequence alignment methods, which do not perform well. Considering there are alignment tools for machine translation, automatic alignment for English should be easier right?

  • 1
    $\begingroup$ Could you say what bio-sequence alignment methods you have tried? $\endgroup$ – AHusain Sep 11 '18 at 2:45
  • $\begingroup$ Actually not bio-sequence in particular, I have tried SequenceMatcher from difflib. I have also tried another one but had error running aligner. Do you have any experience on this? $\endgroup$ – Blue482 Sep 11 '18 at 9:52
  • 1
    $\begingroup$ It looks like the most common mistakes are insertions of spaces and deletions of punctuation and replacing punctuation by spaces. So in particular s(" ","-") should be less of a penalty than s("a","p") in Smith-Waterman. That is the sort of parameter setting I wanted you to include in what you had already tried. $\endgroup$ – AHusain Sep 11 '18 at 14:17

Use kaldi's align-text which align two sentences using Levenshtein distance.

Code: https://github.com/kaldi-asr/kaldi/blob/master/src/bin/align-text.cc


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.