5
$\begingroup$

Please refer to page $93$, Pattern Recognition book by Bishop. This question is related to partial derivative of a term, the proof refers to appendix C.19 which states: $$\frac{\partial({a^{T}x})}{\partial{x}} = \frac{\partial({x^{T}a})}{\partial{x}} = a$$ and then moves on to take partial of normal probability distribution w.r.t. $\mu$ to get (eqn 2.120): $$\frac{\partial(x_{n} - \mu)^{T}\Sigma^{-1}(x_{n} - \mu)}{\partial{\mu}} = (x_{n} - \mu)$$

I am unable to see how we get the term. I tried checking elsewhere, but the explanation "consider $(x_{n}-\mu)$ to be scalar and $(x_{n}-\mu)^{T}$ as vector" does not resonate, as per my understanding both are vectors (as $x_{n}$ is vector) - it's their dot product that results in scalar.

I tried solving it step-by-step: Expand the term, ignoring $\Sigma^{-1}$ (contains no $\mu$ term) which gives: $$\frac{\partial{(x_{n}^{T}x_{n} - x_{n}^{T}\mu - \mu^{T}x_{n} + \mu^{T}\mu)}}{\partial{\mu}}$$ As per my understanding $\frac{\partial}{\partial{\mu}}$ of: $x_{n}^{T}x_{n} = 0$; of $-x_{n}^{T}\mu = -\mu^{T}x_{n} = -\mu$ but, not sure about $\mu^{T}\mu$

Given the context, I request help on partials :

  1. Is $\frac{\partial}{\partial{\mu}}$[ $x_{n}^{T}x_{n} = 0$, $(-x_{n}^{T}\mu = -\mu^{T}x_{n}) = -\mu$]
  2. $\frac{\partial}{\partial{\mu}}(\mu^{T}\mu)$
  3. I have some understanding of calculus, so guide to minimum that'll help me derive equation 2.120.

Magnus & Neudecker will take time (450+ pages) and so I may have to gallop over another partial on page 94 $$ \frac{\partial{(x_{n}-\mu)^{T}\Sigma^{-1}(x_{n}-\mu)}}{{\partial{\Sigma}}}$$

$\endgroup$

1 Answer 1

2
$\begingroup$
  1. $$\frac{\partial}{\partial \mu} (x_n^Tx_n)=0, $$$$\frac{\partial}{\partial \mu}( -\mu^Tx_n)=-x_n$$

  2. $$\frac{\partial }{\partial \mu}(\mu^T\mu) = 2\mu$$

  3. Note that if $A$ is symmetric, then $$\frac{\partial }{\partial y}(y^TAy)=2Ay.$$

\begin{align}\frac{\partial }{\partial \mu}\left(-\frac12 (x_n-\mu)^T\Sigma^{-1}(x_n-\mu) \right)&=-\frac12\frac{\partial }{\partial \mu}\left( (\mu-x_n)^T\Sigma^{-1}(\mu-x_n) \right)\\ &=-\frac12\left(2\Sigma^{-1}(\mu-x_n) \right)\\ &= \Sigma^{-1}(x_n-\mu)\end{align}

$\endgroup$
4
  • $\begingroup$ Got it! subject to my (mis)understanding - a $y^{T}y$ can be simply considered as $y^{2}$. And it $\frac{\partial}{\partial{y}}$ looks lot derivative chain. Correct me. $\endgroup$ Jun 12, 2019 at 2:00
  • $\begingroup$ well, that is the special case in $1$ dimension (of which it better be consistent), be careful when you write $y^2$ as vector squared is not well defined. $\endgroup$ Jun 12, 2019 at 2:02
  • $\begingroup$ ..pardon additional comment, for some some reason $\frac{\partial(a^{T}y)}{\partial{y}}$ turns $a^{T}$ to $a$. Isn't $a$ constant and so come out un-tampered by derivative - Bishop C.19. Regards. $\endgroup$ Jun 12, 2019 at 2:07
  • $\begingroup$ $a^Ty=y^Ta$. Note that there are two conventions in writing down the gradient, the column convention and the row convention. I use the column convention, hence I would write $a$. $\endgroup$ Jun 12, 2019 at 3:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.