0
$\begingroup$

Please refer section 2.3 (page 86-87) in Pattern Recognition and Machine Learning - Bishop

$$\mathit{N}(\mathbf{x}|\mathbf{\mu}, \Sigma)$$ where

$ \mathbf{x} = \begin{bmatrix} x_{a} \\ x_{b} \end{bmatrix} $, $ \mathbf{\mu} = \begin{bmatrix} \mu_{a} \\ \mu_{b} \end{bmatrix} $, $ \mathbf{\Sigma} = \begin{bmatrix} \Sigma_{aa} && \Sigma_{ab}\\ \Sigma_{ba} && \Sigma_{bb} \end{bmatrix} $

The equation below expresses the Quadratic term of exponent in the Bivariate Gaussian probability distribution (ref. book eq. 2.70; ignoring $-\frac{1}{2}$):

$$(x - \mu)^{T}\Sigma^{-1}(x - \mu) = (x_a - \mu_a)^{T}\Lambda_{aa}(x_a - \mu_a) +(x_a - \mu_a)^{T}\Lambda_{ab}(x_b - \mu_b) +(x_b - \mu_b)^{T}\Lambda_{ba}(x_a - \mu_a) +(x_b - \mu_b)^{T}\Lambda_{aa}(x_b - \mu_b)$$

Author states "..conditional distribution $p(x_{a} | x_{b})$ can be evaluated from the the joint distribution $p(x) = p(x_{a}, x_{b})$ by fixing $x_b$ to the observed value and normalizing the resulting expression to obtain the valid probability distribution over $x_{a}$ .." and then filters out only the linear terms with $x_{a}$ and $x_{a}^{T}$ component. The result is presented in equation 2.74 as:

$$x_{a}^{T}\{\Lambda_{aa}\mu_{a} - \Lambda_{ab}(x_b - \mu_b)\}$$

I follow the steps, and see that above is due to $x_{a}^{T}$, $x_{a}$ in the first 2 terms of equation 2.70 - where "term(s)" refer to the components in 2.70 added using $+$ operator.

However, I seem to be getting two additions elements in $x_a$, in excess of the those present in 2.74 - due to the linear $x_a$ factor in $3^{rd}$ term (of equation 2.70). Excess term(s): $$ (x_{b} - \mu_{b})^{T}\Lambda_{ba}x_a $$

Where am I going wrong? Please guide.

Note: I am taking the $x_a$ too to be linear component, whereas 2.74 is (seemingly) considering only the $x_{a}^{T}$ as linear term. Seems like I am losing track somewhere in the middle (perhaps in equation 2.71 - related extract, below):

$$ -\frac{1}{2}(x - \mu)^{T}\Sigma^{-1}(x - \mu) = -\frac{1}{2}{x^{T}\Sigma^{-1}x} + {x^{T}\Sigma^{-1}\mu} + constant \ldots eq. 2.71$$

...where 'const' denotes terms which are independent of ${x}$, and we have made use of the symmetry of $\Sigma$.

$\endgroup$
0
$\begingroup$

\begin{align}&-\frac12(x - \mu)^{T}\Sigma^{-1}(x - \mu) \\&= -\frac12(x_a - \mu_a)^{T}\Lambda_{aa}(x_a - \mu_a) \tag{1} \\&-\frac12(x_a - \mu_a)^{T}\Lambda_{ab}(x_b - \mu_b) \tag{2} \\&-\frac12(x_b - \mu_b)^{T}\Lambda_{ba}(x_a - \mu_a) \tag{3} \\&-\frac12(x_b - \mu_b)^{T}\Lambda_{aa}(x_b - \mu_b) \tag{4}\end{align}

Now let's focus on $(1)$, the linear part in $x_a$ are

$$-\frac12 (-x_a^T\Lambda_{aa}\mu_a - \mu_a^T\Lambda_{aa}x_a)=x_a^T\Lambda_{aa}\mu_a \tag{5}$$

Now focus on $(2)$, the linear part in $x_a$ is

$$-\frac12x_a^T\Lambda_{ab}(x_b-\mu_b) \tag{6}$$

Now focus on $(3)$, the linear part in $x_a$ is

$$-\frac12(x_b-\mu_b)^T\Lambda_{ba}x_a=-\frac12x_a^T\Lambda_{ab}(x_b-\mu_b) \tag{7}$$

There is no linear term of $x_a$ in $(4)$,

Adding $(5)$ to $(7)$, we have

\begin{align}x_a^T\Lambda_{aa}\mu_a-\frac12x_a^T\Lambda_{ab}(x_b - \mu_b)-\frac12x_a^T\Lambda_{ab}(x_b - \mu_b)&=x_a^T\Lambda_{aa}\mu_a-x_a^T\Lambda_{ab}(x_b - \mu_b)\\ &=x_a^T(\Lambda_{aa}\mu_a -\Lambda_{ab}(x_b-\mu_b))\end{align}

$\endgroup$
5
  • $\begingroup$ Almost there! but missing something elementary - in eqn(5) and eqn(7), we're considering $A^{T}$ = A, (and moving ahead with calculation). This is something I am unable to recall...need help there $\endgroup$ – Continue2Learn Jun 10 '19 at 16:03
  • $\begingroup$ $\Sigma$ is a positive definite symmetric matrix right? Hence $\Lambda$ is symmetrical as well. $\endgroup$ – Siong Thye Goh Jun 10 '19 at 16:10
  • $\begingroup$ and please, the reason for $x_{a}$ = $x_{a}^{T}$ and $\mu_{a}$ = $\mu_{a}^{T}$. Thanks $\endgroup$ – Continue2Learn Jun 10 '19 at 16:17
  • $\begingroup$ $x_a = x_a^T$ is not true. What I used is suppose $x^TAy=y^TA^Tx$ since both sides are scalars. $\endgroup$ – Siong Thye Goh Jun 10 '19 at 16:26
  • $\begingroup$ Thanks. Closing the answer with my understanding that $(x_{a}\Lambda_{aa}\mu_{a}^{T}) = {(x_{a}^{T}\Lambda_{aa}\mu_{a})}^{T}$ (eq. (5)). And, L.H.S. = R.H.S. "Only because" the operation equals scalar values. Note: above reason, in current context, does not mean $\(AB)\^{T} = A\^{T}B\^{T}$ as Transpose operation is conducted on only one term of equation 5, LHS. $\endgroup$ – Continue2Learn Jun 10 '19 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.