I am working on implementing IBM Model 1. I have a parallel corpus of some 20,00,000 sentences (English to Dutch). Also, the sentences of the two docs are already aligned. Aim is to translate a Dutch sentence to English and vice-versa. The code I am using for generating the alignments is:
A=pair_sent.split() //To split English sentence B=pair_sent.split() //To split Dutch sentence trips.append([zip(A, p) for p in product(B, repeat=len(A))])
Now, there are pair sentences with unequal number of words (like 10 in English and 14 in its Dutch Translation). Our professor told us that we should use NULLs or drop a word. But I don't understand how to do that? Where to insert NULL and how to choose which word to drop?