2
$\begingroup$

I have an English document, which is preprocessed into two versions. I want to align words or sentences from these two versions of the document. A simple example is as below:

I don't want to go there. My e-mail address ok.

should be aligned with

I do n't want to go there my email address ok

Or the tokens from the first sentence are aligned correctly with the tokens in the 2nd sentence. I have tried some sequence alignment methods, which do not perform well. Considering there are alignment tools for machine translation, automatic alignment for English should be easier right?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Could you say what bio-sequence alignment methods you have tried? $\endgroup$
    – AHusain
    Commented Sep 11, 2018 at 2:45
  • $\begingroup$ Actually not bio-sequence in particular, I have tried SequenceMatcher from difflib. I have also tried another one but had error running aligner. Do you have any experience on this? $\endgroup$
    – Blue482
    Commented Sep 11, 2018 at 9:52
  • 1
    $\begingroup$ It looks like the most common mistakes are insertions of spaces and deletions of punctuation and replacing punctuation by spaces. So in particular s(" ","-") should be less of a penalty than s("a","p") in Smith-Waterman. That is the sort of parameter setting I wanted you to include in what you had already tried. $\endgroup$
    – AHusain
    Commented Sep 11, 2018 at 14:17

1 Answer 1

Reset to default
1
$\begingroup$

Use kaldi's align-text which align two sentences using Levenshtein distance.

Code: https://github.com/kaldi-asr/kaldi/blob/master/src/bin/align-text.cc

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.