1
$\begingroup$

There is a homework question for a course I am self studying (not a student) that is:

let our $n \times d$-dimensional data vectors be denoted by $x_1,\ldots,x_n$ and let $R$ be a $d \times d$ rotation matrix. For simplicity, you may assume that the $x_t$'s have been centered at $0$. Let $$x_t' = Rx_t + v$$ where $v$ is some fixed translation. Forming a second dataset. Now, for any $K$ we pick, let us use PCA on each of the two data sets to obtain $K$-dimensional projections $y_1,\ldots ,y_n$ and $y_1',\ldots,y_n'$, respectively.

Write down a relationship between the two PCA projection matrices $W$ and $W'$ in terms of the rotation matrix $R$ and the translation vector $v$. Explain mathematically how you arrived at this answer.

My answer is basically that while for the untransformed dataset we have that $W$ is the top $k$ eigenvectors of $S[\mu] (S[\mu]$ is the covariance matrix of the untransformed dataset centered at $\mu (\mu$ is $0$ for untransformed dataset)) that $W'$ will be the top $k$ eigenvectors of the matrix $(R^T) \times S[R^T(\mu-v)] R$. ($R^T$ is $R$ transposed). I did this by applying the transformation in the definition of the covariance matrix $S'$ to find a relation between $S'$ and $S$.

The goal of the problem is to show the rotational and translational invariance of PCA. Can anyone give an explanation for this?

$\endgroup$
  • $\begingroup$ You can use LaTeX with $ notation on this site $\endgroup$ – Martin Thoma Jan 11 '19 at 7:17
  • $\begingroup$ Please note: PCA is NOT rotationally invariant. Only if you rotate all data, but it's pretty hard to find a method which is not rotationally invariant in that sense. Usually, people call an (image recognition) algorithm rotationally invariant, if you can rotate single images and not influence the output $\endgroup$ – Martin Thoma Jan 11 '19 at 7:20
1
$\begingroup$

Let $e$ be the all-one column vector of length $n$.

Let $X\in \mathbb{R}^{n\times d}$ be the original matrix.

and $X'$ be the transformed matrix, that is we have $X'=XR^T+ev^T$.

Let's first compute the transformed mean assuming the mean of $X$ is $0$.

$$\frac{e^TX'}{n}=\frac{e^TXR'}{n}+\frac{e^Tev^T}{n}=v^T$$

To compute the singular vectors of the original matrix, theoretically, we can compute the eigenvalues of $$X^TX=UDU^T$$

And we can extract $W$ from $U$.

To compute the PCA for the transformed data

We have $$(X'-ev^T)^T(X'-ev^T)=(XR^T)^T(XR^T)=R^TX^TXR^T=RUDU^TR^T$$

Now, we can extract $W'$ from $RU$.

That is $$W'=RW.$$

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.