Unable to understand Kernel Ridge regression

Question

I am trying to read kernel ridge regression from this link

But , I am unable to get the intution behind the derivation.

Can anyone please help me ?

Answer 1

Equation (4) simply gives us an identity (without proof):

$$(P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} = PB^T (BPB^T + R)^{-1}.$$

Let's check that it does indeed hold. The left hand side equals

\begin{align*} (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} &= \left(\left((P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1}\right)^{-1}\right)^{-1}\\ &= \left(RB^{-T}(P^{-1} + B^TR^{-1}B)\right)^{-1}\\ &= \left(RB^{-T}P^{-1} + B\right)^{-1}. \end{align*} The right hand side equals \begin{align*} PB^T (BPB^T + R)^{-1} &= \left(\left(PB^T (BPB^T + R)^{-1}\right)^{-1}\right)^{-1}\\ &= \left((BPB^T + R)B^{-T}P^{-1} \right)^{-1}\\ &= \left(B + RB^{-T}P^{-1} \right)^{-1}. \end{align*} So indeed, the equality holds.

If you take the right hand side of equation (3) $$\left(\lambda\mathbf{I} + \sum_i \mathbf{x}_i \mathbf{x}_i^T\right)^{-1}\left(\sum_j y_j \mathbf{x}_j\right),$$ replace $\mathbf{x}_i$ with $\phi_i$ and rewrite it as $$\left(\lambda\mathbf{I} + \Phi\Phi^T\right)^{-1}\left(\Phi\mathbf{y}\right),$$ then it matches the left hand side of the identity in equation (4) (right-multiplied by $\mathbf{y}$), where $$P^{-1} = \lambda^{-1}\mathbf{I}$$ $$B^T = \Phi$$ $$R = \mathbf{I}.$$

Therefore, we can rewrite the right hand side of equation (4) as

$$\lambda^{-1}\mathbf{I}\Phi(\Phi\lambda^{-1}\Phi + \mathbf{I})^{-1}\mathbf{y}.$$ As $\lambda$ is a scalar, we can rewrite this as $$\Phi(\Phi\Phi + \lambda\mathbf{I})^{-1}\mathbf{y}.$$