# Calculation of PCA

Consider the following data set:

Now, we need to calculate the principal component analysis for this data. Here are the eigenvalues and eigenvectors calculated for the covariance matrix of this data:

So the principal component is:

Now, when I have tried to do so by hand, I have found that the eigenvalues are 1.28 and 0.0492 (which are identical to the above solution). Surely, the principal component corresponds to the eigenvalue = 1.28. However, when I tried to solve for the eigenvector, the solution was [[0] [0]] as the augmented matrix of BX=0 Was [[1 0 0] [0 1 0]]. So where is the problem here? And also how can I find the transformed data after I calculate the principle component?

Edit: the covariance matrix is

• I am not exactly sure how to help out without trying a few different thing because I'm not exactly sure how you have defined Covariance matrix and principal component. There are a lot of non-standard, but still often used, definitions for both of these objects. If you could clarify, I might be able to help you out. For example, you ask about the transformed data, but also refer to the calculated principal component, which as far as I know it, is the transformed data. Commented Feb 27, 2021 at 6:15
• I have added the definition of the covariance matrix in the question. For the transformed data, after finding the principal components we need to reduce the dimensionality of the data using the calculated principal components. This is the transformed data. Commented Feb 27, 2021 at 16:12
• to calculate the principal components (i.e. dimension reduced data), you can decompose the original data matrix as $D=[X,Y] = U \Sigma V^*$ via an SVD. You can reconstruct the data matrix $D$ and capture some percent of the variance in the data by taking only $r$ first columns in $U$, $r$ first rows in $V^*$ and $r$ first singular values in $\Sigma$ (a diagonal matrix). Then the principal components, or as you call the transformed data, is given by $U^*D = \Sigma V^*$. Either lhs or rhs suffice. You can also compute the PC by taking eigendecomposition of $D^TD$. Commented Feb 27, 2021 at 19:14

Before you start anything, it can be helpful to conduct some exploratory data analysis (EDA) so that you can get a general sense of the data you are working with and what you are going to be doing.

Here is a plot of the data:

It looks like there is a strong linear relationship in this dataset, so this is a strong candidate for where using PCA will likely be able to capture a large portion of the variance using only a single feature.

## Method 1: By hand (sort of)

Step 1

Start by converting your dataset into a matrix.

$$\begin{array}{c|lcr} & \text{X} & \text{Y} \\ \hline 1 & 2.5 & 2.4\\ 2 & .5 & .7\\ 3 & 2.2 & 2.9\\ 4 & 1.9 & 2.2\\ 5 & 3.1 & 3.0\\ 6 & 2.3 & 2.7\\ 7 & 2.0 & 1.6\\ 8 & 1.0 & 1.1\\ 9 & 1.5 & 1.6\\ 10 & 1.1 & .9\\ \end{array} \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \textbf{A} = \begin{bmatrix} 2.5 & 2.4 \\ .5 & .7\\ 2.2 & 2.9\\ 1.9 & 2.2\\ 3.1 & 3.0\\ 2.3 & 2.7\\ 2.0 & 1.6\\ 1.0 & 1.1\\ 1.5 & 1.6\\ 1.1 & .9\\ \end{bmatrix}$$

Step 2

Calculate the Covariance matrix.

Recall, that this is specified by:

$$\begin{bmatrix} Var(x) & Con(x,y) \\ Cov(x,y) & Var(y)\\ \end{bmatrix}$$

In order to calculate these values, we need to get the means $$\bar{\text{X}}$$ and $$\bar{\text{Y}}$$

$$\bar{\text{X}} = { {2.5 \ + \ .5 \ + \ 2.2 \ + \ 1.9 \ + \ 3.1 \ + \ 2.3 \ + \ 2.0 \ + \ 1.0 \ + \ 1.5 \ + \ 1.1} \over {10} } = \fbox{1.81}$$

$$\bar{\text{Y}} = { {2.4 \ + \ .7 \ + \ 2.9 \ + \ 2.2 \ + \ 3.0 \ + \ 2.7 \ + \ 1.6 \ + \ 1.1 \ + \ 1.6 \ + \ .9} \over {10} } = \fbox{1.91}$$

We calculate the values for $$Var(X)$$, $$Var(Y)$$, and $$Cov(X,Y)$$ below by plugging in the values to the formulas:

$$Var(X) = { {\sum_{i=1}^n({x_i} - \bar{x})^2} \over {n-1} } = { {5.549} \over {9} } = \fbox{0.616555555556}$$

$$Var(Y) = { {\sum_{i=1}^n({y_i} - \bar{y})^2} \over {n-1} } = { {6.449} \over {9} } = \fbox{0.716555555556}$$

$$Cov(X,Y) = { {\sum_{i=1}^n({x_i} - \bar{x})({y_i} - \bar{y})} \over {n-1} } = { {5.539} \over {9} } = \fbox{.61544444444}$$

Putting these results into the matrix, we get the Covariance matrix:

$$\begin{bmatrix} 0.61655556 & 0.61544444 \\ 0.61544444 & 0.71655556\\ \end{bmatrix}$$

Step 3

Find the Eigen Values by taking the Eigen decomposition of the Covariance matrix.

Using the formula $$A v = \lambda v$$, we can rewrite it as $$(A-\lambda I)v=0$$ and note that this equation will have a solution at $$det(A-\lambda I)=0$$

$$det = \Bigg| \begin{bmatrix} 0.61655556 & 0.61544444 \\ 0.61544444 & 0.71655556\\ \end{bmatrix} - \lambda \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ \end{bmatrix} \Bigg| = 0$$

$$det = \Bigg| \begin{bmatrix} 0.61655556 & 0.61544444 \\ 0.61544444 & 0.71655556\\ \end{bmatrix} - \begin{bmatrix} \lambda & 0 \\ 0 & \lambda \\ \end{bmatrix} \Bigg| = 0$$

$$det = \Bigg| \begin{bmatrix} 0.61655556 - \lambda & 0.61544444 \\ 0.61544444 & 0.71655556 - \lambda \\ \end{bmatrix} \Bigg| = 0$$

$$(0.61655556 - \lambda)(0.71655556 - \lambda) \ - \ (0.61544444)(0.61544444) =$$

$$(\lambda^2 - 1.33311112 \lambda + .441796314567 - .378771858727) =$$

$$(\lambda^2 - 1.33311112 \lambda + .06302445584) =$$

$$\textbf{Eigen Values:} \ \lambda = \fbox{.0490834} \ \text{or} \ \lambda = \fbox{1.2840277}$$

Step 4 (Here is where we cannot really proceed solely by hand)

Find the Eigen Vectors.

For the first Eigen Vector $$v_1$$ for $$\lambda = .0490834$$

Using the formula $$A v_1 = \lambda_1 v_1$$

$$\begin{bmatrix} 0.61655556 - \lambda & 0.61544444 \\ 0.61544444 & 0.71655556 - \lambda \\ \end{bmatrix} \begin{bmatrix} v_{1,1} \\ v_{1,2} \\ \end{bmatrix} = 0$$

$$\begin{bmatrix} 0.61655556 - .0490834 & 0.61544444 \\ 0.61544444 & 0.71655556 - .0490834 \\ \end{bmatrix} \begin{bmatrix} v_{1,1} \\ v_{1,2} \\ \end{bmatrix} = 0$$

$$\begin{bmatrix} .56747216 & 0.61544444 \\ 0.61544444 & .66747216 \\ \end{bmatrix} \begin{bmatrix} v_{1,1} \\ v_{1,2} \\ \end{bmatrix} = 0$$

Note: This is not an easy equation to solve by hand! I would recommend using MATLAB here.

For the second Eigen Vector $$v_2$$ for $$\lambda = 1.2840277$$

Using the formula $$A v_2 = \lambda_1 v_2$$

$$\begin{bmatrix} 0.61655556 - \lambda & 0.61544444 \\ 0.61544444 & 0.71655556 - \lambda \\ \end{bmatrix} \begin{bmatrix} v_{2,1} \\ v_{2,2} \\ \end{bmatrix} = 0$$

$$\begin{bmatrix} 0.61655556 - 1.2840277 & 0.61544444 \\ 0.61544444 & 0.71655556 - 1.2840277 \\ \end{bmatrix} \begin{bmatrix} v_{2,1} \\ v_{2,2} \\ \end{bmatrix} = 0$$

$$\begin{bmatrix} -.667477214 & 0.61544444 \\ 0.61544444 & -.61655554 \\ \end{bmatrix} \begin{bmatrix} v_{2,1} \\ v_{2,2} \\ \end{bmatrix} = 0$$

Note: This is not an easy equation to solve by hand! I would recommend using MATLAB here.

Since it is not an easy way to solve for either of these Eigen vectors by hand, I recommend using MATLAB.

MATLAB code:

A = [0.61655556 0.61544444; 0.61544444 0.71655556]
A

[v,d]=eig(A)

v


Output:

-0.735178655741955  0.677873398313764
0.677873398313764   0.735178655741955


## Method 2 (using NumPy)

We can verify the results that we get above by using NumPy.

import numpy as np

# create a numpy array that stores the data matrix
matrix = np.array([[2.5, 2.4], [.5, .7], [2.2, 2.9], [1.9, 2.2], [3.1, 3.0],
[2.3, 2.7],[2.0, 1.6],[1.0, 1.1],[1.5, 1.6], [1.1, .9]])

# calculate the covariance matrix
covariance_matrix = np.cov(matrix[:,0], matrix[:,1])

# create variables to store the Eigen values and Eigen vectors
eigen_values, eigen_vectors = np.linalg.eig(covariance_matrix)


Output:

Eigen Values:

array([0.0490834 , 1.28402771])


Eigen Vectors:

array([[-0.73517866, -0.6778734 ],
[ 0.6778734 , -0.73517866]])


## Method 3 (using scikit-learn)

Using the same matrix variable defined above in the NumPy example:

from sklearn.decomposition import PCA
import pandas as pd

# create and fit a PCA model
pca = PCA(n_components=2)
pca.fit(matrix)

# show the Eigen vector
pca.components_

# show the Eigen values
pca.explained_variance_


Output:

Eigen Vectors:

[[-0.6778734  -0.73517866]
[-0.73517866  0.6778734 ]]


Eigen Values:

[1.28402771 0.0490834 ]


Finally, we can show plot vectors that show the direction and magnitudes of the principal axes. As you can see, the first component is longer because it explains more of the variance. We can also project the data onto the first principal component using only one feature.

Conclusion: In short, while you can calculate a good portion of this problem by hand, once you get to the part about calculating Eigen Vectors, this becomes fairly difficult. At this point, I would recommend using a computational method, such as using MATLAB, NumPy, or scikit-learn.