2
$\begingroup$

This refers to Chapter 2, section 2.2 (page 75) - Pattern Recognition and Machine Learning, Christopher Bishop.

The question is related to calculation of Maximum Likelihood variable:

Maximum likelihood is given as: $$p(D|\mu) = \prod_{n=1}^{N}\prod_{k=1}^{K}\mu_{k}^{x_{nk}} = \prod_{k=1}^{K}\mu_{k}^{m_{k}} \tag{2.29}$$ where $m_{k} = \sum_{n}x_{nk}$ $\implies$ #occurrences of event $k$, $x_k$, over $n$ trials where, k={1,$\ldots$,K} and, n={1,$\ldots$,N}. Also, $\mu_{k}$ being probability, and $D$, the observed data.

Author applies Lagrangian multiplier $\lambda$ and take log-likehood of above equation to get: $$\sum_{k=1}^{K}m_{k}ln{\mu_{k}} + \lambda(\sum_{k=1}^{K}\mu_{k} - 1) \tag{2.31}$$ given the (probability) constraint): $$\sum_{k=1}^{K}\mu_{k} = 1$$

Maximization of eqn. 2.31, w.r.t $\mu_{k}$ and equating it to 0, yields: $$ \mu_{k} = -m_{k}/\lambda \tag{2.32}$$ Author substitutes the constraint in eq 2.32 and finally gets: $$ \mu_k^{ML} = \frac{m_k}{N} \tag{2.33} $$

I am losing it in the last step - my (partial/ wrong) derivation of 2.31 w.r.t. $\mu_{k}$ yields: $$ \sum_{k=1}^{K}\frac{m_{k}}{\mu_{k}} + \lambda\cdot K = 0$$

I am not able to understand:

  1. Where am I going wrong
  2. How to move ahead to get the result (only, if my derivation is correct)
$\endgroup$

1 Answer 1

1
$\begingroup$

The partial derivative you have calculated is incorrect. You are probably confused by the index k in the equation. Consider expanding the Langrangian: $$ L = m_{1}ln\mu _{1}+ m_{2}ln\mu_{2} + ... + m_{K}ln\mu_{K} + \lambda(\mu_{1} + \mu_{2} + ... + \mu_{2} - 1) $$

Now, take the derivative with respect to $ \mu_{1} $

$$ \frac{\partial L}{\partial \mu _{1}} = \frac{m_{1}}{\mu _{1}} + \lambda $$

Then, in general, $$ \frac{\partial L}{\partial \mu _{k}} = \frac{m_{k}}{\mu _{k}} + \lambda $$

Equate it to zero to get $ \mu _{k} = -m_{k} / \lambda $

Now substitute it in the constraint $$ \sum_{k=1}^{K}\mu_{k} = \sum_{k=1}^{K}\frac{-m_{k}}{\lambda} = 1 $$

Then $ \lambda = \sum_{k=1}^{K}-m_{k} = -N $

Finally, you get your answer $$ \mu_{k} = \frac{m_{k}}{N} $$

$\endgroup$
1
  • $\begingroup$ Got it! I was ignoring the fact that $\mu$ = $[\mu_{1}, \mu_{2}, \ldots, \mu_{K} ]^T$ is a vector. $\endgroup$ Commented Jun 19, 2019 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.