0
$\begingroup$

I'm learning the theory and implementation of PCA algorithm in the book 'Mathematics For Machine Learning' and finishing the official tutorial notebook in https://colab.research.google.com/github/mml-book/mml-book.github.io/blob/master/tutorials/tutorial_pca.ipynb.

Taking an encoder-decoder perspective, as the figure encoder-decoder illustration shows, the algorithm could be formulated as a linear, lossy auto-encoder. Specifically, the centered data matrix $X\in \mathbb{R}^{D\times N}$, where $N, D$ are the number of samples and the sample dimension respectively. The encoder part is: $$ Z=B^TX\in \mathbb{R}^{M\times N} $$ where $B\in \mathbb{R}^{D\times M}$ is constructed by the $M$ eigenvectors as columns corresponding to the $M$ largest eigenvalues of covariance matrix $S=\frac{1}{N}XX^T$. And the reconstruction data by the decoder are: $$ \hat{X}=BZ=BB^TX\in \mathbb{R}^{D \times N} $$ So the projection matrix is $P=BB^T$. But in the official implementation/solution of the tutorial in https://colab.research.google.com/github/mml-book/mml-book.github.io/blob/master/tutorials/tutorial_pca.solution.ipynb, it calculates the projection matrix as follows:

projection_matrix = (B @ np.linalg.inv(B.T @ B) @ B.T)

and the reconstruction matrix as:

reconst = (projection_matrix @ X.T)

And this corresponds to calculating the projection matrix as follows: $$ P=B(B^TB)^{-1}B^T $$ instead of $BB^T$. The only difference of the implementation here is that the data matrix is transposed as opposed to the formulation in the book, so $X\in \mathbb{R}^{N\times D}$. So I'm confused by the way the tutorial calculates the projection matrix. Can anyone give some explanations on this?

$\endgroup$

1 Answer 1

0
$\begingroup$

Since B has a D*M dimension then Bt (B transposed) will be of a M * D dimension, P will have a D * D dimension multiplied by X which has a D * N dimension, X¨ will have a D * N dimension, same as X.
Bt @ B is the covariance matrix.I honestly don't know why did they insert it in the middle and change the formulas, but the formulas are valid

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.