# Unable to understand Kernel Ridge regression

But , I am unable to get the intution behind the derivation.

• Which part, specifically, is unclear? – ignoring_gravity May 21 '18 at 8:45
• hi @Lupacante , thanks for your reply. I am specifically unclear about the equation (4) and henceforth. – DukeLover May 21 '18 at 11:42

Equation (4) simply gives us an identity (without proof):

$$(P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} = PB^T (BPB^T + R)^{-1}.$$

Let's check that it does indeed hold. The left hand side equals

\begin{align*} (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} &= \left(\left((P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1}\right)^{-1}\right)^{-1}\\ &= \left(RB^{-T}(P^{-1} + B^TR^{-1}B)\right)^{-1}\\ &= \left(RB^{-T}P^{-1} + B\right)^{-1}. \end{align*} The right hand side equals \begin{align*} PB^T (BPB^T + R)^{-1} &= \left(\left(PB^T (BPB^T + R)^{-1}\right)^{-1}\right)^{-1}\\ &= \left((BPB^T + R)B^{-T}P^{-1} \right)^{-1}\\ &= \left(B + RB^{-T}P^{-1} \right)^{-1}. \end{align*} So indeed, the equality holds.

If you take the right hand side of equation (3) $$\left(\lambda\mathbf{I} + \sum_i \mathbf{x}_i \mathbf{x}_i^T\right)^{-1}\left(\sum_j y_j \mathbf{x}_j\right),$$ replace $\mathbf{x}_i$ with $\phi_i$ and rewrite it as $$\left(\lambda\mathbf{I} + \Phi\Phi^T\right)^{-1}\left(\Phi\mathbf{y}\right),$$ then it matches the left hand side of the identity in equation (4) (right-multiplied by $\mathbf{y}$), where $$P^{-1} = \lambda^{-1}\mathbf{I}$$ $$B^T = \Phi$$ $$R = \mathbf{I}.$$

Therefore, we can rewrite the right hand side of equation (4) as

$$\lambda^{-1}\mathbf{I}\Phi(\Phi\lambda^{-1}\Phi + \mathbf{I})^{-1}\mathbf{y}.$$ As $\lambda$ is a scalar, we can rewrite this as $$\Phi(\Phi\Phi + \lambda\mathbf{I})^{-1}\mathbf{y}.$$

• Hi , I am not able to understand from where this identity is coming ? – DukeLover May 21 '18 at 13:28
• @DukeLover I've edited my answer accordingly - is it clear now? – ignoring_gravity May 22 '18 at 8:30