I'm currently trying to implement this paper, but am struggling to understand some of the math here. I'm pretty sure I understand how to implement the E-step, but for the M-step, I'm confused on how to compute the M-step. It says just before section 3.1 that $p_1(x, z; \theta_1) = p(e)p(a, f|e; \theta_1)$, and then the same for $p_2$ but with $e$ and $f$ swapped. The second part of this makes sense to me, but what is $p(e)$ or $p(f)$? From my understanding, $e, f$ are sentences in the bi-text. So how would we compute the probability of a sentence?

It says earlier that $p(e)$ and $p(f)$ are arbitrary distributions that don't affect the optimization problem, but then how do we compute $p_1(x, z; \theta_1)$?



You are right that $p(e)$ is the probability of the English sentence. Estimating the probability of a sentence is achieved by a language model.

This kind of machine translation model is known as the noisy channel model. The noisy channel model says that given a french sentence $f$, its best English translation is

$$e^* = \arg\max_{e\in E} p(e)p(f|e)$$

In this equation the $p(e)$ is the language model. Back in the era of IBM models (which are built upon the noisy channel approach), it is usually an n-gram based language model, calculated as (assuming bigram) $$p(e_1e_2...e_n)=p(e_1|<s>)p(e_2|e_1)p(e_3|e_2)...p(</s>|e_n)$$

And $p(f|e)$ is the translation model where you need to use the EM algorithm to solve. Inside the EM algorithm you do not update the language model parameters, so yes, $p(e)$ and $p(f)$ don't affect the optimization problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.