5
$\begingroup$

Please refer to page $93$, Pattern Recognition book by Bishop. This question is related to partial derivative of a term, the proof refers to appendix C.19 which states: $$\frac{\partial({a^{T}x})}{\partial{x}} = \frac{\partial({x^{T}a})}{\partial{x}} = a$$ and then moves on to take partial of normal probability distribution w.r.t. $\mu$ to get (eqn 2.120): $$\frac{\partial(x_{n} - \mu)^{T}\Sigma^{-1}(x_{n} - \mu)}{\partial{\mu}} = (x_{n} - \mu)$$

I am unable to see how we get the term. I tried checking elsewhere, but the explanation "consider $(x_{n}-\mu)$ to be scalar and $(x_{n}-\mu)^{T}$ as vector" does not resonate, as per my understanding both are vectors (as $x_{n}$ is vector) - it's their dot product that results in scalar.

I tried solving it step-by-step: Expand the term, ignoring $\Sigma^{-1}$ (contains no $\mu$ term) which gives: $$\frac{\partial{(x_{n}^{T}x_{n} - x_{n}^{T}\mu - \mu^{T}x_{n} + \mu^{T}\mu)}}{\partial{\mu}}$$ As per my understanding $\frac{\partial}{\partial{\mu}}$ of: $x_{n}^{T}x_{n} = 0$; of $-x_{n}^{T}\mu = -\mu^{T}x_{n} = -\mu$ but, not sure about $\mu^{T}\mu$

Given the context, I request help on partials :

  1. Is $\frac{\partial}{\partial{\mu}}$[ $x_{n}^{T}x_{n} = 0$, $(-x_{n}^{T}\mu = -\mu^{T}x_{n}) = -\mu$]
  2. $\frac{\partial}{\partial{\mu}}(\mu^{T}\mu)$
  3. I have some understanding of calculus, so guide to minimum that'll help me derive equation 2.120.

Magnus & Neudecker will take time (450+ pages) and so I may have to gallop over another partial on page 94 $$ \frac{\partial{(x_{n}-\mu)^{T}\Sigma^{-1}(x_{n}-\mu)}}{{\partial{\Sigma}}}$$

$\endgroup$
2
$\begingroup$
  1. $$\frac{\partial}{\partial \mu} (x_n^Tx_n)=0, $$$$\frac{\partial}{\partial \mu}( -\mu^Tx_n)=-x_n$$

  2. $$\frac{\partial }{\partial \mu}(\mu^T\mu) = 2\mu$$

  3. Note that if $A$ is symmetric, then $$\frac{\partial }{\partial y}(y^TAy)=2Ay.$$

\begin{align}\frac{\partial }{\partial \mu}\left(-\frac12 (x_n-\mu)^T\Sigma^{-1}(x_n-\mu) \right)&=-\frac12\frac{\partial }{\partial \mu}\left( (\mu-x_n)^T\Sigma^{-1}(\mu-x_n) \right)\\ &=-\frac12\left(2\Sigma^{-1}(\mu-x_n) \right)\\ &= \Sigma^{-1}(x_n-\mu)\end{align}

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Got it! subject to my (mis)understanding - a $y^{T}y$ can be simply considered as $y^{2}$. And it $\frac{\partial}{\partial{y}}$ looks lot derivative chain. Correct me. $\endgroup$ – Continue2Learn Jun 12 '19 at 2:00
  • $\begingroup$ well, that is the special case in $1$ dimension (of which it better be consistent), be careful when you write $y^2$ as vector squared is not well defined. $\endgroup$ – Siong Thye Goh Jun 12 '19 at 2:02
  • $\begingroup$ ..pardon additional comment, for some some reason $\frac{\partial(a^{T}y)}{\partial{y}}$ turns $a^{T}$ to $a$. Isn't $a$ constant and so come out un-tampered by derivative - Bishop C.19. Regards. $\endgroup$ – Continue2Learn Jun 12 '19 at 2:07
  • $\begingroup$ $a^Ty=y^Ta$. Note that there are two conventions in writing down the gradient, the column convention and the row convention. I use the column convention, hence I would write $a$. $\endgroup$ – Siong Thye Goh Jun 12 '19 at 3:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.