# Maximum Entropy modelling - likelihood equation

I am trying to understand maximum entropy modelling and I came across log likelihood equation of the empirical distribution, which I did not quite understand, which also eventually turns out to be equal to the dual function we get when trying to maximise the entropy with constraints using Lagrange's multipliers

$$L_\widetilde{p}(p) \equiv log \prod\limits_{x,y}p(y|x)^{\widetilde{p}(x,y)} = \sum\limits_{x,y}\widetilde{p}(x,y)\log{p(y|x)}$$

Where,

• $y$ is the outcome produced by a random process
• $x$ is the feature influencing the outcome $y$
• $L_\widetilde{p}(p)$ is the log likelihood
• $\widetilde{p}$ is the empirical distribution of training data
• $p(y|x)$ is the model
• $\widetilde{p}(x,y) \equiv \frac{1}{N} \times \text{number of times that } (x,y) \text{ occurs in the sample}$

Can someone please explain how $p(y|x)$ is raised to the power of $\widetilde{p}(x,y)$ in the log likelihood equation mentioned above? Instead, shouldn't $p(x|y)$ be raised to the power of number of times that $(x,y)$ occurs in the sample.

I went through this reference tutorial on max entropy.

## 2 Answers

Let $X_1, ..., X_n$ have a discrete join probability distribution $P(x \mid \theta)$ where $\vec{x} = (x_1, ... , x_n)$ is a vector in the sample and $\theta$ is a parameter from parameter space. We will call $P(x \mid \theta)$ the likelihood function when it is a function of $\theta$. By the ML principle we must maximize the likelihood function over some training set. Assuming that the samples are i.i.d then ML is maximize by:

$theta^{*} = argmax \sum_{i=1}^{m} log(P(x_i \mid \theta))$
The above part should be review.

Let $\chi$ be the set $\{a_1, ... , a_n\}$ and let $P(a \mid \phi)$ be the probability of drawing $a_i$. Finally let $x_1, ... , x_n$ be the sequence of the drawn and be i.i.d based on probability P. $f(a)$ is the measurement of the draws i.e. $f(a) = | \{i : x_i = a\} |$.

The empirical distribution is defined by $\hat{P}(a) = \frac{f(a)}{\sum_{a \in \chi}f(\alpha)} = \frac{f(a)}{m}$.

The joint probability $P(x_1, .. x_m | \theta) = \prod_{i=1}^{m}p(x_{i} | \phi)^{f(a)}$

As you can see here Training data, your tutorial and I have defined our emperical distribution similiary and obtained the same result.

TLDR, it's just a matter of notation.

I'd merely add a comment if I could ...

What is your interpretation of p(y|x)? The usual expression is "the probability of y given x", i.e. the dependent variable corresponding to the independent variable in the probability distribution. This is the function you want to maximize, not the inverse function.

As a side note, there are two ways that the entropy form of the equation on the right can be used. This can be added to a cost function as a regularization kernel, with a modulating coefficient; or it can be the primary term in the cost function -- the entropy to be maximized -- with constraints from the physical system introduced with Lagrange multipliers.