0
$\begingroup$

Please refer section 2.3 (page 86-87) in Pattern Recognition and Machine Learning - Bishop

$$\mathit{N}(\mathbf{x}|\mathbf{\mu}, \Sigma)$$ where

$ \mathbf{x} = \begin{bmatrix} x_{a} \\ x_{b} \end{bmatrix} $, $ \mathbf{\mu} = \begin{bmatrix} \mu_{a} \\ \mu_{b} \end{bmatrix} $, $ \mathbf{\Sigma} = \begin{bmatrix} \Sigma_{aa} && \Sigma_{ab}\\ \Sigma_{ba} && \Sigma_{bb} \end{bmatrix} $

The equation below expresses the Quadratic term of exponent in the Bivariate Gaussian probability distribution (ref. book eq. 2.70; ignoring $-\frac{1}{2}$):

$$(x - \mu)^{T}\Sigma^{-1}(x - \mu) = (x_a - \mu_a)^{T}\Lambda_{aa}(x_a - \mu_a) +(x_a - \mu_a)^{T}\Lambda_{ab}(x_b - \mu_b) +(x_b - \mu_b)^{T}\Lambda_{ba}(x_a - \mu_a) +(x_b - \mu_b)^{T}\Lambda_{aa}(x_b - \mu_b)$$

Author states "..conditional distribution $p(x_{a} | x_{b})$ can be evaluated from the the joint distribution $p(x) = p(x_{a}, x_{b})$ by fixing $x_b$ to the observed value and normalizing the resulting expression to obtain the valid probability distribution over $x_{a}$ .." and then filters out only the linear terms with $x_{a}$ and $x_{a}^{T}$ component. The result is presented in equation 2.74 as:

$$x_{a}^{T}\{\Lambda_{aa}\mu_{a} - \Lambda_{ab}(x_b - \mu_b)\}$$

I follow the steps, and see that above is due to $x_{a}^{T}$, $x_{a}$ in the first 2 terms of equation 2.70 - where "term(s)" refer to the components in 2.70 added using $+$ operator.

However, I seem to be getting two additions elements in $x_a$, in excess of the those present in 2.74 - due to the linear $x_a$ factor in $3^{rd}$ term (of equation 2.70). Excess term(s): $$ (x_{b} - \mu_{b})^{T}\Lambda_{ba}x_a $$

Where am I going wrong? Please guide.

Note: I am taking the $x_a$ too to be linear component, whereas 2.74 is (seemingly) considering only the $x_{a}^{T}$ as linear term. Seems like I am losing track somewhere in the middle (perhaps in equation 2.71 - related extract, below):

$$ -\frac{1}{2}(x - \mu)^{T}\Sigma^{-1}(x - \mu) = -\frac{1}{2}{x^{T}\Sigma^{-1}x} + {x^{T}\Sigma^{-1}\mu} + constant \ldots eq. 2.71$$

...where 'const' denotes terms which are independent of ${x}$, and we have made use of the symmetry of $\Sigma$.

$\endgroup$

1 Answer 1

0
$\begingroup$

\begin{align}&-\frac12(x - \mu)^{T}\Sigma^{-1}(x - \mu) \\&= -\frac12(x_a - \mu_a)^{T}\Lambda_{aa}(x_a - \mu_a) \tag{1} \\&-\frac12(x_a - \mu_a)^{T}\Lambda_{ab}(x_b - \mu_b) \tag{2} \\&-\frac12(x_b - \mu_b)^{T}\Lambda_{ba}(x_a - \mu_a) \tag{3} \\&-\frac12(x_b - \mu_b)^{T}\Lambda_{aa}(x_b - \mu_b) \tag{4}\end{align}

Now let's focus on $(1)$, the linear part in $x_a$ are

$$-\frac12 (-x_a^T\Lambda_{aa}\mu_a - \mu_a^T\Lambda_{aa}x_a)=x_a^T\Lambda_{aa}\mu_a \tag{5}$$

Now focus on $(2)$, the linear part in $x_a$ is

$$-\frac12x_a^T\Lambda_{ab}(x_b-\mu_b) \tag{6}$$

Now focus on $(3)$, the linear part in $x_a$ is

$$-\frac12(x_b-\mu_b)^T\Lambda_{ba}x_a=-\frac12x_a^T\Lambda_{ab}(x_b-\mu_b) \tag{7}$$

There is no linear term of $x_a$ in $(4)$,

Adding $(5)$ to $(7)$, we have

\begin{align}x_a^T\Lambda_{aa}\mu_a-\frac12x_a^T\Lambda_{ab}(x_b - \mu_b)-\frac12x_a^T\Lambda_{ab}(x_b - \mu_b)&=x_a^T\Lambda_{aa}\mu_a-x_a^T\Lambda_{ab}(x_b - \mu_b)\\ &=x_a^T(\Lambda_{aa}\mu_a -\Lambda_{ab}(x_b-\mu_b))\end{align}

$\endgroup$
5
  • $\begingroup$ Almost there! but missing something elementary - in eqn(5) and eqn(7), we're considering $A^{T}$ = A, (and moving ahead with calculation). This is something I am unable to recall...need help there $\endgroup$ Jun 10, 2019 at 16:03
  • $\begingroup$ $\Sigma$ is a positive definite symmetric matrix right? Hence $\Lambda$ is symmetrical as well. $\endgroup$ Jun 10, 2019 at 16:10
  • $\begingroup$ and please, the reason for $x_{a}$ = $x_{a}^{T}$ and $\mu_{a}$ = $\mu_{a}^{T}$. Thanks $\endgroup$ Jun 10, 2019 at 16:17
  • $\begingroup$ $x_a = x_a^T$ is not true. What I used is suppose $x^TAy=y^TA^Tx$ since both sides are scalars. $\endgroup$ Jun 10, 2019 at 16:26
  • $\begingroup$ Thanks. Closing the answer with my understanding that $(x_{a}\Lambda_{aa}\mu_{a}^{T}) = {(x_{a}^{T}\Lambda_{aa}\mu_{a})}^{T}$ (eq. (5)). And, L.H.S. = R.H.S. "Only because" the operation equals scalar values. Note: above reason, in current context, does not mean $\(AB)\^{T} = A\^{T}B\^{T}$ as Transpose operation is conducted on only one term of equation 5, LHS. $\endgroup$ Jun 10, 2019 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.