I am trying to understand maximum entropy modelling and I came across log likelihood equation of the empirical distribution, which I did not quite understand, which also eventually turns out to be equal to the dual function we get when trying to maximise the entropy with constraints using Lagrange's multipliers

$$ L_\widetilde{p}(p) \equiv log \prod\limits_{x,y}p(y|x)^{\widetilde{p}(x,y)} = \sum\limits_{x,y}\widetilde{p}(x,y)\log{p(y|x)}$$


  • $y$ is the outcome produced by a random process
  • $x$ is the feature influencing the outcome $y$
  • $L_\widetilde{p}(p)$ is the log likelihood
  • $\widetilde{p}$ is the empirical distribution of training data
  • $p(y|x)$ is the model
  • $\widetilde{p}(x,y) \equiv \frac{1}{N} \times \text{number of times that } (x,y) \text{ occurs in the sample} $

Can someone please explain how $p(y|x)$ is raised to the power of $\widetilde{p}(x,y)$ in the log likelihood equation mentioned above? Instead, shouldn't $p(x|y)$ be raised to the power of number of times that $(x,y)$ occurs in the sample.

I went through this reference tutorial on max entropy.


Let $X_1, ..., X_n$ have a discrete join probability distribution $P(x \mid \theta)$ where $\vec{x} = (x_1, ... , x_n)$ is a vector in the sample and $\theta$ is a parameter from parameter space. We will call $P(x \mid \theta)$ the likelihood function when it is a function of $\theta$. By the ML principle we must maximize the likelihood function over some training set. Assuming that the samples are i.i.d then ML is maximize by:

$theta^{*} = argmax \sum_{i=1}^{m} log(P(x_i \mid \theta))$
The above part should be review.

Let $\chi$ be the set $\{a_1, ... , a_n\}$ and let $P(a \mid \phi)$ be the probability of drawing $a_i$. Finally let $x_1, ... , x_n$ be the sequence of the drawn and be i.i.d based on probability P. $f(a)$ is the measurement of the draws i.e. $f(a) = | \{i : x_i = a\} |$.

The empirical distribution is defined by $\hat{P}(a) = \frac{f(a)}{\sum_{a \in \chi}f(\alpha)} = \frac{f(a)}{m}$.

The joint probability $P(x_1, .. x_m | \theta) = \prod_{i=1}^{m}p(x_{i} | \phi)^{f(a)}$

As you can see here Training data, your tutorial and I have defined our emperical distribution similiary and obtained the same result.

TLDR, it's just a matter of notation.


I'd merely add a comment if I could ...

What is your interpretation of p(y|x)? The usual expression is "the probability of y given x", i.e. the dependent variable corresponding to the independent variable in the probability distribution. This is the function you want to maximize, not the inverse function.

As a side note, there are two ways that the entropy form of the equation on the right can be used. This can be added to a cost function as a regularization kernel, with a modulating coefficient; or it can be the primary term in the cost function -- the entropy to be maximized -- with constraints from the physical system introduced with Lagrange multipliers.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.