Source - Murphy, 12.3

Heuristic for assessing applicability of PCA.

Let the empirical covariance matrix Σ have eigenvalues λ1≥λ2≥···≥λd>0, with mean λ. Explain why the variance of the eigen values, σ2=(1/d)∑(λi−λ)2 is a good measure of whether or not PCA would be useful for analysing the data (the higher the value of σ2 the more useful PCA)


1 Answer 1


What is important is that the covariance matrix eigenvalues represent the variance explained by each component of the PCA.

PCA can be considered useful when it allows to reduce the dimension without removing too much variance from the dataset; that is, when there are a lot of low eigenvalues. As the sum of the eigenvalues is fixed because it is the total variance of the dataset (this is a property of the eigenvalues decomposition of a matrix: the trace of a matrix is equal to the sum of its eigenvalues), low eigenvalues for some eigenvectors means that there are other high eigenvalues to compensate; that is a high variance in the eigenvalues.

In a more mathematical way, if we denote by $\sigma_{tot}^2$ the total variance over the dataset (not the variance of eigenvalues), we know that the mean of eigenvalues will be $\frac{\sigma_{tot}^2}{n}$. As all eigenvalues are non-negative (property of the covariance matrix), and based on the trace conservation property, we deduce that all eigenvalues are lower or equal to $\sigma_{tot}^2$. Then, the convexity of the square function implies that maximum variance among eigenvalues is reached when one of the eigenvalues is equal to $\sigma_{tot}^2$, and all other are null. Hence, eigenvalues maximum variance is reached when the first component wears the total dataset variance, which is obviously the optimal PCA case (all features reduced to a single dimension).

Oppositely, the case with minimal variance of eigenvalues is the one where all eigenvalue are equal, meaning that there is no particular "principal" component (the projection of the dataset onto any direction gives the exact same variance). If I'm not mistaken in the math, this is only possible when the covariance matrix is a multiple of the identity matrix, meaning the dataset is already completely uncorrelated. In that case, PCA is simply useless.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.