1
$\begingroup$

I am trying to read kernel ridge regression from this link

But , I am unable to get the intution behind the derivation.

Can anyone please help me ?

$\endgroup$
2
  • $\begingroup$ Which part, specifically, is unclear? $\endgroup$ May 21, 2018 at 8:45
  • $\begingroup$ hi @Lupacante , thanks for your reply. I am specifically unclear about the equation (4) and henceforth. $\endgroup$
    – DukeLover
    May 21, 2018 at 11:42

1 Answer 1

1
$\begingroup$

Equation (4) simply gives us an identity (without proof):

$$(P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} = PB^T (BPB^T + R)^{-1}.$$

Let's check that it does indeed hold. The left hand side equals

\begin{align*} (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} &= \left(\left((P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1}\right)^{-1}\right)^{-1}\\ &= \left(RB^{-T}(P^{-1} + B^TR^{-1}B)\right)^{-1}\\ &= \left(RB^{-T}P^{-1} + B\right)^{-1}. \end{align*} The right hand side equals \begin{align*} PB^T (BPB^T + R)^{-1} &= \left(\left(PB^T (BPB^T + R)^{-1}\right)^{-1}\right)^{-1}\\ &= \left((BPB^T + R)B^{-T}P^{-1} \right)^{-1}\\ &= \left(B + RB^{-T}P^{-1} \right)^{-1}. \end{align*} So indeed, the equality holds.

If you take the right hand side of equation (3) $$\left(\lambda\mathbf{I} + \sum_i \mathbf{x}_i \mathbf{x}_i^T\right)^{-1}\left(\sum_j y_j \mathbf{x}_j\right),$$ replace $\mathbf{x}_i$ with $\phi_i$ and rewrite it as $$\left(\lambda\mathbf{I} + \Phi\Phi^T\right)^{-1}\left(\Phi\mathbf{y}\right),$$ then it matches the left hand side of the identity in equation (4) (right-multiplied by $\mathbf{y}$), where $$P^{-1} = \lambda^{-1}\mathbf{I}$$ $$B^T = \Phi$$ $$R = \mathbf{I}.$$

Therefore, we can rewrite the right hand side of equation (4) as

$$\lambda^{-1}\mathbf{I}\Phi(\Phi\lambda^{-1}\Phi + \mathbf{I})^{-1}\mathbf{y}.$$ As $\lambda$ is a scalar, we can rewrite this as $$\Phi(\Phi\Phi + \lambda\mathbf{I})^{-1}\mathbf{y}.$$

$\endgroup$
2
  • $\begingroup$ Hi , I am not able to understand from where this identity is coming ? $\endgroup$
    – DukeLover
    May 21, 2018 at 13:28
  • $\begingroup$ @DukeLover I've edited my answer accordingly - is it clear now? $\endgroup$ May 22, 2018 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.