I am trying to read kernel ridge regression from this link

But , I am unable to get the intution behind the derivation.

Can anyone please help me ?

  • $\begingroup$ Which part, specifically, is unclear? $\endgroup$ – marco_gorelli May 21 '18 at 8:45
  • $\begingroup$ hi @Lupacante , thanks for your reply. I am specifically unclear about the equation (4) and henceforth. $\endgroup$ – DukeLover May 21 '18 at 11:42

Equation (4) simply gives us an identity (without proof):

$$(P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} = PB^T (BPB^T + R)^{-1}.$$

Let's check that it does indeed hold. The left hand side equals

\begin{align*} (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} &= \left(\left((P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1}\right)^{-1}\right)^{-1}\\ &= \left(RB^{-T}(P^{-1} + B^TR^{-1}B)\right)^{-1}\\ &= \left(RB^{-T}P^{-1} + B\right)^{-1}. \end{align*} The right hand side equals \begin{align*} PB^T (BPB^T + R)^{-1} &= \left(\left(PB^T (BPB^T + R)^{-1}\right)^{-1}\right)^{-1}\\ &= \left((BPB^T + R)B^{-T}P^{-1} \right)^{-1}\\ &= \left(B + RB^{-T}P^{-1} \right)^{-1}. \end{align*} So indeed, the equality holds.

If you take the right hand side of equation (3) $$\left(\lambda\mathbf{I} + \sum_i \mathbf{x}_i \mathbf{x}_i^T\right)^{-1}\left(\sum_j y_j \mathbf{x}_j\right),$$ replace $\mathbf{x}_i$ with $\phi_i$ and rewrite it as $$\left(\lambda\mathbf{I} + \Phi\Phi^T\right)^{-1}\left(\Phi\mathbf{y}\right),$$ then it matches the left hand side of the identity in equation (4) (right-multiplied by $\mathbf{y}$), where $$P^{-1} = \lambda^{-1}\mathbf{I}$$ $$B^T = \Phi$$ $$R = \mathbf{I}.$$

Therefore, we can rewrite the right hand side of equation (4) as

$$\lambda^{-1}\mathbf{I}\Phi(\Phi\lambda^{-1}\Phi + \mathbf{I})^{-1}\mathbf{y}.$$ As $\lambda$ is a scalar, we can rewrite this as $$\Phi(\Phi\Phi + \lambda\mathbf{I})^{-1}\mathbf{y}.$$

| improve this answer | |
  • $\begingroup$ Hi , I am not able to understand from where this identity is coming ? $\endgroup$ – DukeLover May 21 '18 at 13:28
  • $\begingroup$ @DukeLover I've edited my answer accordingly - is it clear now? $\endgroup$ – marco_gorelli May 22 '18 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.