0
$\begingroup$

In pca, we convert predictors into principal components for dimensionality reduction. My assumption is every principal component is a eigen vector with eigen value as sum of squared distance of orthogonal projection of data points or vectors on it.. Is my assumption correct. If not pls clarify

$\endgroup$
1
$\begingroup$

Your assumption is correct. However, some additional details need to be specified e.g. eigenvector of what ? What characterizes the principal component eigenvector i.e. what criteria must it satisfy ?

As explained in the authoritative reference book by Bishop, the Principal Component Analysis (PCA) provides a set of mutually orthogonal directions, called principal components, which maximize the variance of the projections of the data points, when it is projected successively on respective directions as axes. Here, the underlying assumption is the data points are considered elements of a Vector Space equipped with standard inner product.

Illustrating using first principal component, which say is specified by unit vector, $\mathbf{u_1}$ for a dataset, $\{ \mathbf{x_n} \}, \text{n} = 1, \dots, N$ then

  • Projection of $ \mathbf{x_n} $ = $ \mathbf{u_1}^T \mathbf{x_n} $
  • Mean of Projections, $ \mathbb{E}_{\mathbf{u_1}}(\mathbf{x}) = \mathbf{u_1}^T \bar{\mathbf{x}} $ where $ \bar{\mathbf{x}} = \frac{1}{N} \sum^N_{n=1} \mathbf{x_n} $
  • Variance of Projections, $ Var_{\mathbf{u_1}}(\mathbf{x}) = \frac{1}{N} \sum^N_{n=1} ( \mathbf{u_1}^T \mathbf{x_n} - \mathbf{u_1}^T \bar{\mathbf{x}} )^2 = \mathbf{u_1}^T \mathbf{S} \mathbf{u_1} $ where $\mathbf{S}$ is the Data Covariance matrix given by $ \mathbf{S} = \frac{1}{N} \sum^N_{n=1} ( \mathbf{x_n} - \mathbf{\bar{x}} ) ( \mathbf{x_n} - \mathbf{\bar{x}} )^T $

Maximizing $ Var(\mathbf{x}) $ subject to unit vector constraint, $ \mathbf{u_1}^T \mathbf{u_1} = 1 $, applying Lagrange multiplier we get optimization problem

$ \text{max}_{ \mathbf{u_1}^T \mathbf{u_1} = 1 } \mathbf{u_1}^T \mathbf{S} \mathbf{u_1} = \text{max}_{ \mathbf{u_1} } \mathbf{u_1}^T \mathbf{S} \mathbf{u_1} + \lambda_1 ( \mathbf{u_1}^T \mathbf{u_1} - 1 ) $

At maxima, we get
$ \mathbf{S} \mathbf{u_1} = \lambda_1 \mathbf{u_1} \implies \lambda_1 = \mathbf{u_1}^T \mathbf{S} \mathbf{u_1} = Var_{\mathbf{u_1}}(\mathbf{x}) $

Hence, principal eigenvectors e.g. $\mathbf{u_1}$ are eigenvectors of Covariance matrix, $\mathbf{S}$ with corresponding eigenvalues e.g. $\lambda_1$ as Variance of projections along corresponding eigenvector

| improve this answer | |
$\endgroup$
1
$\begingroup$

Yes, a principal component is defined as the eigenvector of the covariance matrix of the data (zero-centered).

The eigenvector with the largest eigenvalue points in the direction of the greatest variance in the data. The smaller the eigenvalue, the less dispersed data points are in the direction of that eigenvector.

I'd recommend reading on the intuition behind PCA on Wikipedia.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.