I am working on implementing IBM Model 1. I have a parallel corpus of some 20,00,000 sentences (English to Dutch). Also, the sentences of the two docs are already aligned. Aim is to translate a Dutch sentence to English and vice-versa. The code I am using for generating the alignments is:

    A=pair_sent[0].split()  //To split English sentence
    B=pair_sent[1].split()  //To split Dutch sentence
    trips.append([zip(A, p) for p in product(B, repeat=len(A))])

Now, there are pair sentences with unequal number of words (like 10 in English and 14 in its Dutch Translation). Our professor told us that we should use NULLs or drop a word. But I don't understand how to do that? Where to insert NULL and how to choose which word to drop?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.